[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412901#comment-16412901
]
Hive QA commented on HIVE-18825:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12915946/HIVE-18825.08.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 119 failed/errored test(s), 13434 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412885#comment-16412885
]
Hive QA commented on HIVE-18825:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 20 new + 558 unchanged - 17
fixed = 578 total (was 575) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9813/dev-support/hive-personality.sh
|
| git revision | master / 5b222db |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9813/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9813/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9813/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.05.patch,
> HIVE-18825.06.patch, HIVE-18825.07.patch, HIVE-18825.08.patch,
> HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411796#comment-16411796
]
Jesus Camacho Rodriguez commented on HIVE-18825:
Reuploading patch.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.05.patch,
> HIVE-18825.06.patch, HIVE-18825.07.patch, HIVE-18825.08.patch,
> HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409814#comment-16409814
]
Hive QA commented on HIVE-18825:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12915692/HIVE-18825.07.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 343 failed/errored test(s), 13484 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409707#comment-16409707
]
Hive QA commented on HIVE-18825:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 20 new + 558 unchanged - 17
fixed = 578 total (was 575) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9771/dev-support/hive-personality.sh
|
| git revision | master / 80aaa1e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9771/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9771/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9771/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.05.patch,
> HIVE-18825.06.patch, HIVE-18825.07.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409611#comment-16409611
]
Jesus Camacho Rodriguez commented on HIVE-18825:
Uploading rebased patch once again to try to trigger ptests.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.05.patch,
> HIVE-18825.06.patch, HIVE-18825.07.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408730#comment-16408730
]
Eugene Koifman commented on HIVE-18825:
---
+1 patch 6 pending tests
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.05.patch,
> HIVE-18825.06.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408718#comment-16408718
]
Jesus Camacho Rodriguez commented on HIVE-18825:
[~ekoifman], the code that accesses the valid txn list was not there when I
created the patch, it was introduced in HIVE-18751, hence I did not realize
there was any dependency at that level. I have made changes to avoid clearing
the txn list, the rest should work as expected.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.05.patch,
> HIVE-18825.06.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408489#comment-16408489
]
Eugene Koifman commented on HIVE-18825:
---
GenericUDTFGetSplits.createPlanFragment() now calls compileAndRespond(String,
true) which removes ValidTxnList from Conf. But createPlanFragment() then
tries to access it. Is that intentional?
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.05.patch,
> HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405367#comment-16405367
]
Jesus Camacho Rodriguez commented on HIVE-18825:
[~ekoifman], could you take a look? We are planning to push the rest of
materialized view work that relies on this patch by EOW. Thanks
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.05.patch,
> HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398821#comment-16398821
]
Jesus Camacho Rodriguez commented on HIVE-18825:
[~ekoifman], can you review it? I have created HIVE-18948 for the follow-up
work, please feel free to add any comment over there if you think I have missed
anything.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.05.patch,
> HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397992#comment-16397992
]
Hive QA commented on HIVE-18825:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914340/HIVE-18825.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 13015 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397951#comment-16397951
]
Hive QA commented on HIVE-18825:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 20 new + 558 unchanged - 17
fixed = 578 total (was 575) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9626/dev-support/hive-personality.sh
|
| git revision | master / d977a9d |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9626/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9626/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9626/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.05.patch,
> HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397586#comment-16397586
]
Jesus Camacho Rodriguez commented on HIVE-18825:
Uploaded a rebased version of the patch and created HIVE-18948 for the
follow-up work.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.05.patch,
> HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391880#comment-16391880
]
Eugene Koifman commented on HIVE-18825:
---
The Lock manger is entirely based on Read/WriteEntities created by compiler.
If you move lock acquisition to some place before they are available, you
basically need to rewrite the entire logic in the LM that is used to figure out
what to lock. It may be possible but it's an unpredictably large amount of
work which makes pessimistic locking an impractically expensive feature.
Incremental refresh is not prevented by this, though I understand that there
are some possible performance optimizations that are difficult w/o this.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389929#comment-16389929
]
Jesus Camacho Rodriguez commented on HIVE-18825:
bq. Just because some feature doesn't exist yet, it doesn't mean we should make
changes that will make that feature impossible in the future.
I agree with your statement, but I disagree that this change makes the feature
impossible. The proposal is to split the work in two steps. When we implement
concurrency based lock control (or we can do it even before that), we will need
to move the lock acquisition logic before query compilation as part of that
feature work.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389911#comment-16389911
]
Eugene Koifman commented on HIVE-18825:
---
Just because some feature doesn't exist yet, it doesn't mean we should make
changes that will make that feature impossible in the future. For example, we
don't have multi statement transactions fully supported, but i constantly pay
attention to it to make sure it will be possible to finish it.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389849#comment-16389849
]
Jesus Camacho Rodriguez commented on HIVE-18825:
[~ekoifman], why? Which alternative do you propose?
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388925#comment-16388925
]
Eugene Koifman commented on HIVE-18825:
---
I'm strongly against this.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388834#comment-16388834
]
Jesus Camacho Rodriguez commented on HIVE-18825:
I have been checking the code, and indeed changing the position for acquiring
the locks requires a lot of work. Although we extract the read entities while
parsing (and I guess we could extend that logic and extract write entities
too), the logic to extract locks, etc., is tightly linked to the query plan
object.
One question I have is: does the proposed change break any current existing use
case or do we need it only in case we want to support lock based concurrency
control?
If it does not break the current system implementation, I suggest we commit
this fix to unlock MV incremental rebuild, etc., and we create a follow-up JIRA
to move the logic to acquire locks to be executed after parsing and before rest
of compilation.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388481#comment-16388481
]
Eugene Koifman commented on HIVE-18825:
---
The lock manger uses Read/WriteEnity in the QueryPlan to know what to lock.
Those are not there after parsing, so I don't see how that can work w/o
rewriting the LM.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388286#comment-16388286
]
Jesus Camacho Rodriguez commented on HIVE-18825:
bq. Could the predicates you want be inserted at compile time, but bound to
actual values as some post processing after (or at the end of)
Driver.acquireLocks() as currently implemented?
That is somehow similar to parameterized queries, that is quite a lot of work
to implement and falls out of scope I think, at least right now.
Also, I think there is a problem with acquiring the locks after compilation
that we had not realized about beforehand ([~ashutoshc] pointed it out).
Queries use information about available partitions during compilation phase...
What happens when new partitions are added or they are removed between
compilation phase and the moment when locks are acquired and valid txn list is
generated? This might lead to incorrect results if we are not misunderstanding
the whole mechanism.
I think the right fix is to call {{lockAndRespond}} after parsing the query and
before optimization/compilation starts, since at that moment we will have all
the information that we need to proceed. What do you think?
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387145#comment-16387145
]
Eugene Koifman commented on HIVE-18825:
---
I think there is a problem here that I should've thought of earlier.
Right now we lock in the snapshot after lock acquisition. This ordering is
important if we ever want to support lock based concurrency control.
Suppose you have 2 concurrent transactions both running "update T1 set x = x +
1". If we acquire the update lock first, then record the snapshot, then 2nd
txn to get the lock will see the result of write of the previous one. If we
lock the snapshot before acquiring the lock, both transactions may lock in
exactly the same snapshot and locking becomes useless as the 2nd will still
read an "old" snapshot.
Could the predicates you want be inserted at compile time, but bound to actual
values as some post processing after (or at the end of)
{{Driver.acquireLocks()}} as currently implemented?
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386319#comment-16386319
]
Jesus Camacho Rodriguez commented on HIVE-18825:
Fails are unrelated. [~ekoifman], [~sankarh], I have created a PR with latest
patch in https://reviews.apache.org/r/65907/ . Could you take a look? Thanks
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384973#comment-16384973
]
Hive QA commented on HIVE-18825:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12912894/HIVE-18825.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 13059 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384960#comment-16384960
]
Hive QA commented on HIVE-18825:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 18 new + 559 unchanged - 17
fixed = 577 total (was 576) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9470/dev-support/hive-personality.sh
|
| git revision | master / 05d4719 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9470/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9470/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9470/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384859#comment-16384859
]
Jesus Camacho Rodriguez commented on HIVE-18825:
{{compileAndRespond}} is called on its own sometimes, e.g., to retrieve a plan
and use it somewhere else or during testing (no need to keep txn list as we may
call {{compileAndRespond}} again without calling {{run}}) and also as part of
prepare in SQLOperation (need to keep txn list as we call {{run}} after that).
HIVE-18825.04.patch fixes this issue.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384788#comment-16384788
]
Hive QA commented on HIVE-18825:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12912830/HIVE-18825.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 101 failed/errored test(s), 13852 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=249)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[interval_comparison]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_div0]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=103)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[split_sample_out_of_range]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[split_sample_wrong_format]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_join_2]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby_2]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_pruning_2]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_grandparent]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_groupby]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_multiple_cols_in_select]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_corr_multi_rows]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_multi_rows]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_select_aggregate]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_select_distinct]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_subquery_chain_exists]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[temp_table_rename]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[touch2]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_bucketed_column]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_seqfile]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_table_failure3]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_table_failure5]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_collect_set_unsupported]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_array_contains_wrong2]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_assert_true2]
(batchId=94)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384751#comment-16384751
]
Hive QA commented on HIVE-18825:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9460/dev-support/hive-personality.sh
|
| git revision | master / 53980ba |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9460/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9460/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384191#comment-16384191
]
Jesus Camacho Rodriguez commented on HIVE-18825:
HIVE-18825.03.patch cleans valid txn list property for {{compileAndRespond}}
statements (fixes {{TestDbTxnManager2}} failures).
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383978#comment-16383978
]
Jesus Camacho Rodriguez commented on HIVE-18825:
The following test failures seem related:
{code}
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking11
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testDynamicPartitionInsert
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.updateSelectUpdate
{code}
Taking a look right now.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383952#comment-16383952
]
Hive QA commented on HIVE-18825:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12912659/HIVE-18825.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 13024 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=170)
[vector_windowing_expressions.q,tez_union_group_by.q,vector_like_2.q,auto_join1.q,llap_acid.q,sqlmerge.q,tez_dynpart_hashjoin_1.q,schema_evol_orc_acid_part_update_llap_io.q,vector_windowing_gby.q,vectorized_timestamp.q,cbo_subq_exists.q,lateral_view.q,schema_evol_orc_vec_table_llap_io.q,optimize_nullscan.q,vectorization_decimal_date.q,schema_evol_orc_nonvec_table_llap_io.q,udaf_all_keyword.q,tez_self_join.q,vector_partitioned_date_time.q,acid_vectorization_original.q,tez_fsstat.q,vector_mapjoin_reduce.q,join_acid_non_acid.q,empty_join.q,vector_groupby_grouping_window.q,auto_join21.q,tez_input_counters.q,schema_evol_orc_nonvec_part_all_complex_llap_io.q,orc_ppd_timestamp.q,vector_decimal_1.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383863#comment-16383863
]
Hive QA commented on HIVE-18825:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
27s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9442/dev-support/hive-personality.sh
|
| git revision | master / 61e21d6 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9442/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9442/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383425#comment-16383425
]
Hive QA commented on HIVE-18825:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12912659/HIVE-18825.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 13053 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383385#comment-16383385
]
Hive QA commented on HIVE-18825:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9436/dev-support/hive-personality.sh
|
| git revision | master / fffbec0 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9436/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9436/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382655#comment-16382655
]
Jesus Camacho Rodriguez commented on HIVE-18825:
Uploaded HIVE-18825.02.patch addressing [~ekoifman] feedback.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382614#comment-16382614
]
Eugene Koifman commented on HIVE-18825:
---
I think so - if we have no txn, we should not need ValidTxnList for query
processin
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382598#comment-16382598
]
Jesus Camacho Rodriguez commented on HIVE-18825:
[~ekoifman], thanks. It is not about opening the txn after I get the list, the
call to get the valid txn list happens after we might open a transaction in
Driver#compile (L595) (I have not changed that).
We hit the assertion when we have not open a transaction yet at all (for
instance, first line to execute in the test is 'use databasex;'). For safety, I
can keep the assertion, and only create the valid txn list if we have already
opened a transaction in the system. Does it sound reasonable?
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382548#comment-16382548
]
Eugene Koifman commented on HIVE-18825:
---
this assumption is not valid. For one thing if you open the txn after you get
the list, you won't be able to read your writes, .i.e. your own txn will be
above the HWM that is set for you txn. The txn is currently opened right after
parsing - do you need ValidTxnList earlier than that?
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382524#comment-16382524
]
Jesus Camacho Rodriguez commented on HIVE-18825:
Attempt #2: HIVE-18825.01.patch
I have removed the {{assert}} as it seems to me that it is not needed anymore
(you can retrieve the valid txn list without having opened any transaction
yet?) and above test failures are passing now, however this assumption needs
validation. [~sankarh] [~ekoifman]
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381517#comment-16381517
]
Gopal V commented on HIVE-18825:
{code}
java.lang.AssertionError
at
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getValidTxns(DbTxnManager.java:763)
at org.apache.hadoop.hive.ql.Driver.recordValidTxns(Driver.java:1223)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:610)
{code}
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381515#comment-16381515
]
Hive QA commented on HIVE-18825:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12912482/HIVE-18825.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 85 failed/errored test(s), 13447 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381478#comment-16381478
]
Hive QA commented on HIVE-18825:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9414/dev-support/hive-personality.sh
|
| git revision | master / bd6d91f |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9414/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9414/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380727#comment-16380727
]
Jesus Camacho Rodriguez commented on HIVE-18825:
Cc [~sankarh], [~ekoifman]
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW__ID.transactionId < highwatermark and ROW__ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query compilation. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
