subject:"\[jira\] \[Commented\] \(HIVE\-18873\) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result"

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-09-11 Thread Ankit Singhal (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16610378#comment-16610378
 ] 

Ankit Singhal commented on HIVE-18873:
--

Thanks [~vgarg] for taking care of this, Sorry couldn't check the reason for 
the failure as was on vacation from last 2 weeks. 

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.3.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-09-04 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16603954#comment-16603954
 ] 

Hive QA commented on HIVE-18873:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938306/HIVE-18873.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 14923 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13594/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13594/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13594/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12938306 - PreCommit-HIVE-Build

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.3.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-09-04 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16603936#comment-16603936
 ] 

Hive QA commented on HIVE-18873:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
45s{color} | {color:blue} itests/util in master has 52 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
3s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} The patch util passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} ql: The patch generated 0 new + 20 unchanged - 1 
fixed = 20 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m  2s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13594/dev-support/hive-personality.sh
 |
| git revision | master / 33fa62f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/util ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13594/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.3.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this n

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-09-04 Thread Vineet Garg (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16603603#comment-16603603
 ] 

Vineet Garg commented on HIVE-18873:


Uploaded new patch which disable the test for now. Also opened HIVE-20500 to 
investigate that failure

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.3.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-09-04 Thread Prasanth Jayachandran (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16603497#comment-16603497
 ] 

Prasanth Jayachandran commented on HIVE-18873:
--

{quote}Under the assumption that Hive is ensuring the results are accurate 
anyways? (e.g. we'd want to check that all storage handlers work this way)
{quote}
Orc PPD works this way, if orc cannot handle the filter it will drop the filter 
and continue with rest of the filters. The filters will still remain in hive 
operator tree which will get reapplied anyways.

 

If the test failure is unrelated to the patch and is already happening in the 
master, I think we can go ahead and commit the latest patch and track the test 
failure in a separate ticket. 

+1, comment failing test, track it separately. 

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-31 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599165#comment-16599165
 ] 

Hive QA commented on HIVE-18873:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12937936/HIVE-18873.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14916 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.accumulo.predicate.TestAccumuloPredicateHandler.testPushdownComparisonOptNotSupported
 (batchId=198)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[test_teradatabinaryfile] 
(batchId=2)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13536/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13536/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13536/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12937936 - PreCommit-HIVE-Build

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-31 Thread Vineet Garg (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599119#comment-16599119
 ] 

Vineet Garg commented on HIVE-18873:


My understand is that Hive wants to apply whatever predicate it can to storage 
handler, but it also keep the predicate to itself (to apply it after storage 
handler on table scan operator) thus ensuring the correctness in case storage 
handler couldn't handle the predicates. 
I did more investigating and now I am not certain if this assumption is correct 
in all cases. Therefore I am taking back the patch. This will need further 
investigation. I still believe we should get in the current patch and 
investigate this in separate jira.

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-31 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599117#comment-16599117
 ] 

Hive QA commented on HIVE-18873:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
53s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
31s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs 
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
6s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} The patch accumulo-handler passed checkstyle {color} 
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} ql: The patch generated 0 new + 20 unchanged - 1 
fixed = 20 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
40s{color} | {color:red} accumulo-handler generated 1 new + 21 unchanged - 0 
fixed = 22 total (was 21) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 31s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:accumulo-handler |
|  |  Dead store to residual in 
org.apache.hadoop.hive.accumulo.predicate.AccumuloPredicateHandler.getSearchConditions(Configuration)
  At 
AccumuloPredicateHandler.java:org.apache.hadoop.hive.accumulo.predicate.AccumuloPredicateHandler.getSearchConditions(Configuration)
  At AccumuloPredicateHandler.java:[line 394] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13536/dev-support/hive-personality.sh
 |
| git revision | master / a77e046 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13536/yetus/new-findbugs-accumulo-handler.html
 |
| modules | C: accumulo-handler ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13536/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Repo

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-31 Thread Josh Elser (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599033#comment-16599033
 ] 

Josh Elser commented on HIVE-18873:
---

{quote}I am not sure why Hive pushes down such predicates to storage handler.
{quote}
Gotcha. Sorry, I didn't get that the first time.
{quote}hive has already made sure if the predicate couldn't be pushed it will 
apply the whole predicate itself therefore it is safe for Accumulo handler to 
ignore it.
{quote}
Trying to understand... we want to apply whatever predicates we can in a 
storage handler, but continue quietly if the storage handler cannot process 
one? Under the assumption that Hive is ensuring the results are accurate 
anyways? (e.g. we'd want to check that all storage handlers work this way)

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.3.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-31 Thread Vineet Garg (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599009#comment-16599009
 ] 

Vineet Garg commented on HIVE-18873:


I understand that Accumulo can not handle it what I meant was I am not sure why 
Hive pushes down such predicates to storage handler. {{is not null/ is null}} 
was just an example, you can reproduce the issue with following OR query as well
{code:sql}
SELECT u.key, u.country FROM users u where u.country = 'USA' OR u.country = 
'CANADA'" 
{code}

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-31 Thread Josh Elser (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16598689#comment-16598689
 ] 

Josh Elser commented on HIVE-18873:
---

{quote}passes it to storage handler (not sure why since Accumulo handler isn't 
able to handle it).
{quote}
Already answered why in my last comment. There is no such thing as a null row 
to Accumulo – all rows are non-null. Thus, it wasn't something to test. The 
storage handler would just need to be updated to drop this outright.
{quote} Basically it looks like this has been in the system for a while and is 
being exposed now due to these two jiras.
{quote}
Thanks for the explanation on how we got here.

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-30 Thread Vineet Garg (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16598108#comment-16598108
 ] 

Vineet Garg commented on HIVE-18873:


[~elserj] That query was just to demonstrate/reproduce the bug. This query is 
not part of test. Test has join queries for which Hive automatically generate 
is not null filter on join keys and passes it to storage handler (not sure why 
since Accumulo handler isn't able to handle it).

It was probably not caught earlier because hive.optimize.index.filter was off 
by default (therefore hive wasn't pushing the filter to storage handler). 
hive.optimize.index.filter was turned on by HIVE-12342 but then HIVE-15680 
caused hive to not push the filter to storage handler.
Basically it looks like this has been in the system for a while and is being 
exposed now due to these two jiras.

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-30 Thread Josh Elser (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16598062#comment-16598062
 ] 

Josh Elser commented on HIVE-18873:
---

{code:java}
SELECT u.key, u.country FROM users u where u.key is not null;{code}
This query doesn't really make any sense for an HBase or Accumulo because you 
cannot have a null Key (in their bigtable data model). Where did this query 
come from in the first place? Would rather not see all of TestAccumuloCliDriver 
ignored (and just this one query).

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-30 Thread Vineet Garg (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16598037#comment-16598037
 ] 

Vineet Garg commented on HIVE-18873:


Because this is an existing issue with Accumulo I believe it could be 
investigated in separate JIRA. We should disable the test, get the current 
patch in and open new JIRA for fixing it. What do you think [~prasanth_j]?

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-30 Thread Vineet Garg (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16598030#comment-16598030
 ] 

Vineet Garg commented on HIVE-18873:


I looked into the test failure (TestAccumuloCliDriver). Query {{SELECT u.key, 
u.country, c.name, c.key FROM users u JOIN countries c ON (u.country = c.key)}} 
throws following exception:
{noformat}
java.lang.RuntimeException: Unexpected residual predicate: key is not null
at 
org.apache.hadoop.hive.accumulo.predicate.AccumuloPredicateHandler.getSearchConditions(AccumuloPredicateHandler.java:396)
at 
org.apache.hadoop.hive.accumulo.predicate.AccumuloPredicateHandler.getIterators(AccumuloPredicateHandler.java:315)
at 
org.apache.hadoop.hive.accumulo.mr.HiveAccumuloTableInputFormat.getSplits(HiveAccumuloTableInputFormat.java:146)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:506)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:752)
at 
org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:341)
at 
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:332)
at 
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:203)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:571)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
at 
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:571)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:562)
at 
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:423)
at 
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:149)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:210)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2727)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2379)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2049)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1749)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1743)
at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
at 
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1347)
at 
org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1321)
at 
org.apache.hadoop.hive.cli.control.CoreAccumuloCliDriver.runTest(CoreAccumuloCliDriver.java:113)
at 
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
at 
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver(TestAccumuloCliDriver.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunn

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-28 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594656#comment-16594656
 ] 

Hive QA commented on HIVE-18873:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12937364/HIVE-18873.2_reattach.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14897 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_joins] 
(batchId=256)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13500/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13500/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13500/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12937364 - PreCommit-HIVE-Build

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-27 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594620#comment-16594620
 ] 

Hive QA commented on HIVE-18873:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
10s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} ql: The patch generated 0 new + 20 unchanged - 1 
fixed = 20 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m  0s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13500/dev-support/hive-personality.sh
 |
| git revision | master / acc6fa2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13500/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-27 Thread Ankit Singhal (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594510#comment-16594510
 ] 

Ankit Singhal commented on HIVE-18873:
--

bq. Is your local branch up-to date with the master?
uh.. My bad, I was checking on branch-3 only. Just checked on master and it is 
reproducible there, let me see what is causing this test to fail.

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.2_reattach.patch, 
> HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-27 Thread Vineet Garg (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594490#comment-16594490
 ] 

Vineet Garg commented on HIVE-18873:


[~an...@apache.org] Thanks for looking at it. I tried again and it is failing 
for me. Is your local branch up-to date with the master?

{code}
INFO] Running org.apache.hadoop.hive.cli.TestAccumuloCliDriver
[ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 114.45 
s <<< FAILURE! - in org.apache.hadoop.hive.cli.TestAccumuloCliDriver
[ERROR] 
testCliDriver[accumulo_joins](org.apache.hadoop.hive.cli.TestAccumuloCliDriver) 
Time elapsed: 6.627 s <<< FAILURE!
java.lang.AssertionError:
Client execution failed with error code = 1 running "

SELECT u.key, u.country, c.name, c.key FROM users u JOIN countries c
ON (u.country = c.key)" fname=accumulo_joins.q
at org.junit.Assert.fail(Assert.java:88)
at org.apache.hadoop.hive.ql.QTestUtil.failed(QTestUtil.java:2108)
at 
org.apache.hadoop.hive.cli.control.CoreAccumuloCliDriver.runTest(CoreAccumuloCliDriver.java:115)
at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
at 
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver(TestAccumuloCliDriver.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at 
org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:73)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccumuloCliDriver.testCliDriver:59 Client execution failed with 
error code = 1 running "

SELECT u.key, u.country, c.name, c.key FROM users u JOIN countries c
ON (u.country = c.key)" fname=accumulo_joins.q
[INFO]
[ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0
{code}
 

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-27 Thread Ankit Singhal (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594410#comment-16594410
 ] 

Ankit Singhal commented on HIVE-18873:
--

bq. Accumulo test needs to be looked at.
[~vgarg] , I ran Accumulo qtests locally and they are passing for me with the 
fix as well.
{code}
[INFO]  T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hive.cli.TestAccumuloCliDriver
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 148.492 
s - in org.apache.hadoop.hive.cli.TestAccumuloCliDriver
[INFO] Running org.apache.hive.TestDummy
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.042 s 
- in org.apache.hive.TestDummy
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0
[INFO]
{code}

bq. Test results: 
[https://builds.apache.org/job/PreCommit-HIVE-Build/13437/testReport]
Sorry for being late, testReport seems to be purged by now. Let me re-attach 
the patch for debugging.



> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-24 Thread Vineet Garg (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592121#comment-16592121
 ] 

Vineet Garg commented on HIVE-18873:


Accumulo test needs to be looked at. cc [~an...@apache.org], [~prasanth_j] 
[~elserj]

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-24 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591421#comment-16591421
 ] 

Hive QA commented on HIVE-18873:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12936884/HIVE-18873.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14896 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_joins] 
(batchId=256)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/13437/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13437/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13437/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12936884 - PreCommit-HIVE-Build

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-24 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591355#comment-16591355
 ] 

Hive QA commented on HIVE-18873:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
59s{color} | {color:blue} ql in master has 2308 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} ql: The patch generated 0 new + 20 unchanged - 1 
fixed = 20 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m  0s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13437/dev-support/hive-personality.sh
 |
| git revision | master / 9a5f1c2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-13437/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HIVE-18873.2.patch, HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-08-22 Thread Vineet Garg (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589464#comment-16589464
 ] 

Vineet Garg commented on HIVE-18873:


[~an...@apache.org] Can you upload rebased patch? With new policy patch now 
need green run for tests.

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-06-29 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527765#comment-16527765
 ] 

Hive QA commented on HIVE-18873:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913744/HIVE-18873.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14630 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_joins] 
(batchId=253)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/12246/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12246/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12246/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12913744 - PreCommit-HIVE-Build

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-06-29 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527674#comment-16527674
 ] 

Hive QA commented on HIVE-18873:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
5s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} ql: The patch generated 0 new + 23 unchanged - 1 
fixed = 23 total (was 24) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m  1s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12246/dev-support/hive-personality.sh
 |
| git revision | master / 2b0cb07 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-12246/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-04-02 Thread Ankit Singhal (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16422184#comment-16422184
 ] 

Ankit Singhal commented on HIVE-18873:
--

bq. Sorry for lagging behind on this one. Patch looks good to me. +1, pending 
tests.
Thanks [~prasanth_j], It seems test failures are not related to this, can we 
have this committed now?

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-03-29 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419607#comment-16419607
 ] 

Hive QA commented on HIVE-18873:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913744/HIVE-18873.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 81 failed/errored test(s), 13454 tests 
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=93)

[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,nopart_load.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,authorization_insert_noinspriv.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_f

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-03-29 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419514#comment-16419514
 ] 

Hive QA commented on HIVE-18873:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
37s{color} | {color:green} ql: The patch generated 0 new + 16 unchanged - 1 
fixed = 16 total (was 17) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
15s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m  4s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9906/dev-support/hive-personality.sh
 |
| git revision | master / 14bb998 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9906/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9906/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-03-27 Thread Prasanth Jayachandran (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416484#comment-16416484
 ] 

Prasanth Jayachandran commented on HIVE-18873:
--

Sorry for lagging behind on this one. Patch looks good to me. +1, pending tests.

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-03-13 Thread Ankit Singhal (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396760#comment-16396760
 ] 

Ankit Singhal commented on HIVE-18873:
--

ping [~prasanth_j], any help on this ticket would be greatly appreciated. 

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-03-09 Thread Ankit Singhal (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392656#comment-16392656
 ] 

Ankit Singhal commented on HIVE-18873:
--

bq. IIRC, storage handlers use hive.optimize.ppd.storage to push filters down 
to storage handlers. Maybe we can restrict this only when 
hive.optimize.index.filters is used so that storage handlers don't get impacted 
by HIVE-15680.
[~prasanth_j], How about skipping this check when storage handlers implement 
HiveStoragePredicateHandler? 

 AFAICS, Actually, the storage handlers were receiving the right predicate for 
respective alias while creating the input splits. So once the split is created, 
we don't need to push down filter again during recordReader as InputFormat 
should be carrying all required info through their custom InputSplit. (though 
it seems not true for HiveHBaseTableInputFormat only).
 

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-03-08 Thread Josh Elser (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392041#comment-16392041
 ] 

Josh Elser commented on HIVE-18873:
---

{quote}The original issue in HIVE-15680 is when there are 2 table aliases we 
push only one filter to job conf which results in incorrect result. How does 
phoenix or accumulo deal with multiple aliases?
{quote}
Honestly, I'm not sure. The distinction between the engine is a bit obtuse down 
in StorageHandler API, iirc. I know Ankit and Sergey are looking into this.
{quote} IIRC, storage handlers use hive.optimize.ppd.storage to push filters 
down to storage handlers. Maybe we can restrict this only when 
hive.optimize.index.filters is used so that storage handlers don't get impacted 
by HIVE-15680.
{quote}
I think that would make sense. At least my take, the PPD not happening for 
storage handlers removes what was a pretty nice feature of  storage handlers 
outright. Us all being outsiders to the problems you're trying to solve in Hive 
make it hard for us to give good suggestions :)

The quick response was appreciated (even though I then let it sit for two days).

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-03-06 Thread Prasanth Jayachandran (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388330#comment-16388330
 ] 

Prasanth Jayachandran commented on HIVE-18873:
--

The original issue in HIVE-15680 is when there are 2 table aliases we push only 
one filter to job conf which results in incorrect result. How does phoenix or 
accumulo deal with multiple aliases? 
IIRC, storage handlers use hive.optimize.ppd.storage to push filters down to 
storage handlers. Maybe we can restrict this only when 
hive.optimize.index.filters is used so that storage handlers don't get impacted 
by HIVE-15680. 

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

2018-03-06 Thread Josh Elser (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388243#comment-16388243
 ] 

Josh Elser commented on HIVE-18873:
---

[~prasanth_j], [~thejas], I'm really worried about the change that went into 
HIVE-15680. It seems to have some drastically unintended consequences. As we've 
seen for two separate StorageHandlers already (Phoenix and Accumulo), suddenly 
not allowing PPD with the MR engine breaks some pretty basic assumptions on 
correctness.

I can understand the desire to move away from supporting MR, but is there 
anything we can do here to make this a less painful transition?

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

35 matches

Site Navigation

Mail list logo

Footer information