[jira] [Commented] (HIVE-14836) Test the predicate pushing down support for Parquet vectorization read path

[Xinli Shang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807321#comment-16807321
 ] 

Xinli Shang commented on HIVE-14836:


It seems merging errors. Is it something you are going to fix? 

> Test the predicate pushing down support for Parquet vectorization read path
> ---
>
> Key: HIVE-14836
> URL: https://issues.apache.org/jira/browse/HIVE-14836
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-14836.patch
>
>
> We should add more UT test for predicate pushing down support for Parquet 
> vectorization read path.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-14836) Test the predicate pushing down support for Parquet vectorization read path

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807309#comment-16807309
 ] 

Hive QA commented on HIVE-14836:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887112/HIVE-14836.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16803/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16803/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16803/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-04-02 00:04:02.234
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16803/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-04-02 00:04:02.237
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
   34d2bda..2111c01  master -> origin/master
   d32d9d3..70767b5  branch-3   -> origin/branch-3
+ git reset --hard HEAD
HEAD is now at 34d2bda HIVE-21518: GenericUDFOPNotEqualNS does not run in LLAP 
(Jason Dere, reviewed by Vineet Garg)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 2111c01 HIVE-21537: Scalar query rewrite could be improved to 
not generate an extra join if subquery is guaranteed to produce atmost one row 
(Vineet Garg, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-04-02 00:04:04.790
+ rm -rf ../yetus_PreCommit-HIVE-Build-16803
+ mkdir ../yetus_PreCommit-HIVE-Build-16803
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16803
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16803/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: patch failed: 
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetRowGroupFilter.java:18
Falling back to three-way merge...
Applied patch to 
'ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetRowGroupFilter.java'
 with conflicts.
error: patch failed: 
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java:21
Falling back to three-way merge...
Applied patch to 
'ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java'
 with conflicts.
error: patch failed: 
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java:19
Falling back to three-way merge...
Applied patch to 
'ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java'
 with conflicts.
Going to apply patch with: git apply -p0
error: patch failed: 
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetRowGroupFilter.java:18
Falling back to three-way merge...
Applied patch to 
'ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetRowGroupFilter.java'
 with conflicts.
error: patch failed: 
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java:21
Falling back to three-way merge...
Applied patch to 
'ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java'
 with conflicts.
error: patch failed: 
ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java:19
Falling back to three-way merge...
Applied patch to 
'ql/src/test/org/apache/hadoop/hive/ql/io/parquet/VectorizedColumnReaderTestBase.java'
 with conflicts.
U 

[jira] [Commented] (HIVE-14836) Test the predicate pushing down support for Parquet vectorization read path

[Xinli Shang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806928#comment-16806928
 ] 

Xinli Shang commented on HIVE-14836:


Hi [~Ferd] , is this task done? 

> Test the predicate pushing down support for Parquet vectorization read path
> ---
>
> Key: HIVE-14836
> URL: https://issues.apache.org/jira/browse/HIVE-14836
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-14836.patch
>
>
> We should add more UT test for predicate pushing down support for Parquet 
> vectorization read path.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-14836) Test the predicate pushing down support for Parquet vectorization read path

[Ferdinand Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169158#comment-16169158
 ] 

Ferdinand Xu commented on HIVE-14836:
-

Hi [~xuefuz], after further investigation, I found that the vectorization 
reader supports  predicate pushing down when constructing the Parquet file 
reader. So the work left is about enabling some unit tests covering predicate 
pushing down.

> Test the predicate pushing down support for Parquet vectorization read path
> ---
>
> Key: HIVE-14836
> URL: https://issues.apache.org/jira/browse/HIVE-14836
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
>  Labels: pull-request-available
> Attachments: HIVE-14836.patch
>
>
> We should add more UT test for predicate pushing down support for Parquet 
> vectorization read path.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14836) Test the predicate pushing down support for Parquet vectorization read path

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168207#comment-16168207
 ] 

Xuefu Zhang commented on HIVE-14836:


Hi [~Ferd], thanks for your patch. I'm a little confused. The JIRA title and 
the patch itself seem about adding tests, but the JIRA description suggests 
some feature. Could you clarify a little bit? Thanks.

> Test the predicate pushing down support for Parquet vectorization read path
> ---
>
> Key: HIVE-14836
> URL: https://issues.apache.org/jira/browse/HIVE-14836
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
>  Labels: pull-request-available
> Attachments: HIVE-14836.patch
>
>
> Currently we filter blocks using Predict pushing down. We should support it 
> in page reader as well to improve its efficiency. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14836) Test the predicate pushing down support for Parquet vectorization read path

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166855#comment-16166855
 ] 

Hive QA commented on HIVE-14836:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12887112/HIVE-14836.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 11041 tests 
executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=230)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_char] 
(batchId=9)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_queries] 
(batchId=95)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
 (batchId=156)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_table_failure2]
 (batchId=89)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=234)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=215)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01 
(batchId=215)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6814/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6814/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6814/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12887112 - PreCommit-HIVE-Build

> Test the predicate pushing down support for Parquet vectorization read path
> ---
>
> Key: HIVE-14836
> URL: https://issues.apache.org/jira/browse/HIVE-14836
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
>  Labels: pull-request-available
> Attachments: HIVE-14836.patch
>
>
> Currently we filter blocks using Predict pushing down. We should support it 
> in page reader as well to improve its efficiency. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14836) Test the predicate pushing down support for Parquet vectorization read path

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166337#comment-16166337
 ] 

ASF GitHub Bot commented on HIVE-14836:
---

GitHub user winningsix opened a pull request:

https://github.com/apache/hive/pull/251

HIVE-14836: Test the predicate pushing down support for Parquet vecto…

…rization read path

Add more unit test for Predicate pushing down for Parquet Vectorization.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/winningsix/hive HIVE-14836

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/251.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #251


commit d6a0fc65bea35db9f557bca923838a8bfd7266dc
Author: Ferdinand Xu 
Date:   2017-09-14T13:50:57Z

HIVE-14836: Test the predicate pushing down support for Parquet 
vectorization read path




> Test the predicate pushing down support for Parquet vectorization read path
> ---
>
> Key: HIVE-14836
> URL: https://issues.apache.org/jira/browse/HIVE-14836
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
>  Labels: pull-request-available
> Attachments: HIVE-14836.patch
>
>
> Currently we filter blocks using Predict pushing down. We should support it 
> in page reader as well to improve its efficiency. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)