[jira] [Commented] (HIVE-15056) Support index shifting for struct fields

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625955#comment-15625955
 ] 

ASF GitHub Bot commented on HIVE-15056:
---

Github user sunchao closed the pull request at:

https://github.com/apache/hive/pull/108


> Support index shifting for struct fields
> 
>
> Key: HIVE-15056
> URL: https://issues.apache.org/jira/browse/HIVE-15056
> Project: Hive
>  Issue Type: Sub-task
>  Components: File Formats, Physical Optimizer, 
> Serializers/Deserializers
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 2.2.0
>
> Attachments: HIVE-15056.1.patch, HIVE-15056.2.patch
>
>
> In HIVE-13873, the following case doesn't work:
> {code}
> select s.c from tbl
> {code}
> where {{tbl}} is of schema:
> {code}
> a int
> s struct
> {code}
> This is because currently we generate a "pruned" schema (in terms of 
> {{GroupType}}) for Parquet reader to scan the data. However, on the Hive side 
> the object inspector still uses the original schema. In particular, in this 
> case for {{s.c}} the data returned by Parquet reader is in index 0, but the 
> object inspector tries to read it in index 1. Therefore, in correct result 
> will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15056) Support index shifting for struct fields

[Chao Sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625704#comment-15625704
 ] 

Chao Sun commented on HIVE-15056:
-

Sorry it was my mistake. Thanks [~leftylev] for noticing it and [~Ferd] for 
fixing it.

> Support index shifting for struct fields
> 
>
> Key: HIVE-15056
> URL: https://issues.apache.org/jira/browse/HIVE-15056
> Project: Hive
>  Issue Type: Sub-task
>  Components: File Formats, Physical Optimizer, 
> Serializers/Deserializers
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 2.2.0
>
> Attachments: HIVE-15056.1.patch, HIVE-15056.2.patch
>
>
> In HIVE-13873, the following case doesn't work:
> {code}
> select s.c from tbl
> {code}
> where {{tbl}} is of schema:
> {code}
> a int
> s struct
> {code}
> This is because currently we generate a "pruned" schema (in terms of 
> {{GroupType}}) for Parquet reader to scan the data. However, on the Hive side 
> the object inspector still uses the original schema. In particular, in this 
> case for {{s.c}} the data returned by Parquet reader is in index 0, but the 
> object inspector tries to read it in index 1. Therefore, in correct result 
> will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15056) Support index shifting for struct fields

[Ferdinand Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625537#comment-15625537
 ] 

Ferdinand Xu commented on HIVE-15056:
-

The fix version is not correct. It should be committed to master(2.2.0). Thank 
you for finding this.

> Support index shifting for struct fields
> 
>
> Key: HIVE-15056
> URL: https://issues.apache.org/jira/browse/HIVE-15056
> Project: Hive
>  Issue Type: Sub-task
>  Components: File Formats, Physical Optimizer, 
> Serializers/Deserializers
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 2.2.0
>
> Attachments: HIVE-15056.1.patch, HIVE-15056.2.patch
>
>
> In HIVE-13873, the following case doesn't work:
> {code}
> select s.c from tbl
> {code}
> where {{tbl}} is of schema:
> {code}
> a int
> s struct
> {code}
> This is because currently we generate a "pruned" schema (in terms of 
> {{GroupType}}) for Parquet reader to scan the data. However, on the Hive side 
> the object inspector still uses the original schema. In particular, in this 
> case for {{s.c}} the data returned by Parquet reader is in index 0, but the 
> object inspector tries to read it in index 1. Therefore, in correct result 
> will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15056) Support index shifting for struct fields

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624476#comment-15624476
 ] 

Lefty Leverenz commented on HIVE-15056:
---

[~csun], this issue has fix version 2.1.1 but the commit went to master 
(2.2.0), not branch-2.1.  Will it also be committed to branch-2.1?

> Support index shifting for struct fields
> 
>
> Key: HIVE-15056
> URL: https://issues.apache.org/jira/browse/HIVE-15056
> Project: Hive
>  Issue Type: Sub-task
>  Components: File Formats, Physical Optimizer, 
> Serializers/Deserializers
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 2.1.1
>
> Attachments: HIVE-15056.1.patch, HIVE-15056.2.patch
>
>
> In HIVE-13873, the following case doesn't work:
> {code}
> select s.c from tbl
> {code}
> where {{tbl}} is of schema:
> {code}
> a int
> s struct
> {code}
> This is because currently we generate a "pruned" schema (in terms of 
> {{GroupType}}) for Parquet reader to scan the data. However, on the Hive side 
> the object inspector still uses the original schema. In particular, in this 
> case for {{s.c}} the data returned by Parquet reader is in index 0, but the 
> object inspector tries to read it in index 1. Therefore, in correct result 
> will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15056) Support index shifting for struct fields

[Chao Sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622616#comment-15622616
 ] 

Chao Sun commented on HIVE-15056:
-

Committed to the master branch. Thanks [~Ferd] for the review!

> Support index shifting for struct fields
> 
>
> Key: HIVE-15056
> URL: https://issues.apache.org/jira/browse/HIVE-15056
> Project: Hive
>  Issue Type: Sub-task
>  Components: File Formats, Physical Optimizer, 
> Serializers/Deserializers
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 2.1.1
>
> Attachments: HIVE-15056.1.patch, HIVE-15056.2.patch
>
>
> In HIVE-13873, the following case doesn't work:
> {code}
> select s.c from tbl
> {code}
> where {{tbl}} is of schema:
> {code}
> a int
> s struct
> {code}
> This is because currently we generate a "pruned" schema (in terms of 
> {{GroupType}}) for Parquet reader to scan the data. However, on the Hive side 
> the object inspector still uses the original schema. In particular, in this 
> case for {{s.c}} the data returned by Parquet reader is in index 0, but the 
> object inspector tries to read it in index 1. Therefore, in correct result 
> will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15056) Support index shifting for struct fields

[Ferdinand Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15619689#comment-15619689
 ] 

Ferdinand Xu commented on HIVE-15056:
-

Thanks [~csun] for the patch. LGTM +1.

> Support index shifting for struct fields
> 
>
> Key: HIVE-15056
> URL: https://issues.apache.org/jira/browse/HIVE-15056
> Project: Hive
>  Issue Type: Sub-task
>  Components: File Formats, Physical Optimizer, 
> Serializers/Deserializers
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-15056.1.patch, HIVE-15056.2.patch
>
>
> In HIVE-13873, the following case doesn't work:
> {code}
> select s.c from tbl
> {code}
> where {{tbl}} is of schema:
> {code}
> a int
> s struct
> {code}
> This is because currently we generate a "pruned" schema (in terms of 
> {{GroupType}}) for Parquet reader to scan the data. However, on the Hive side 
> the object inspector still uses the original schema. In particular, in this 
> case for {{s.c}} the data returned by Parquet reader is in index 0, but the 
> object inspector tries to read it in index 1. Therefore, in correct result 
> will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15056) Support index shifting for struct fields

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609650#comment-15609650
 ] 

Hive QA commented on HIVE-15056:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12835390/HIVE-15056.2.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10622 tests 
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
 (batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] (batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] (batchId=164)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1827/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1827/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1827/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12835390 - PreCommit-HIVE-Build

> Support index shifting for struct fields
> 
>
> Key: HIVE-15056
> URL: https://issues.apache.org/jira/browse/HIVE-15056
> Project: Hive
>  Issue Type: Sub-task
>  Components: File Formats, Physical Optimizer, 
> Serializers/Deserializers
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-15056.1.patch, HIVE-15056.2.patch
>
>
> In HIVE-13873, the following case doesn't work:
> {code}
> select s.c from tbl
> {code}
> where {{tbl}} is of schema:
> {code}
> a int
> s struct
> {code}
> This is because currently we generate a "pruned" schema (in terms of 
> {{GroupType}}) for Parquet reader to scan the data. However, on the Hive side 
> the object inspector still uses the original schema. In particular, in this 
> case for {{s.c}} the data returned by Parquet reader is in index 0, but the 
> object inspector tries to read it in index 1. Therefore, in correct result 
> will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15056) Support index shifting for struct fields

[Ferdinand Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15608634#comment-15608634
 ] 

Ferdinand Xu commented on HIVE-15056:
-

Looks like the wrong diff file attached.

> Support index shifting for struct fields
> 
>
> Key: HIVE-15056
> URL: https://issues.apache.org/jira/browse/HIVE-15056
> Project: Hive
>  Issue Type: Sub-task
>  Components: File Formats, Physical Optimizer, 
> Serializers/Deserializers
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-15056.1.patch, HIVE-15056.2.patch
>
>
> In HIVE-13873, the following case doesn't work:
> {code}
> select s.c from tbl
> {code}
> where {{tbl}} is of schema:
> {code}
> a int
> s struct
> {code}
> This is because currently we generate a "pruned" schema (in terms of 
> {{GroupType}}) for Parquet reader to scan the data. However, on the Hive side 
> the object inspector still uses the original schema. In particular, in this 
> case for {{s.c}} the data returned by Parquet reader is in index 0, but the 
> object inspector tries to read it in index 1. Therefore, in correct result 
> will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15056) Support index shifting for struct fields

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15607792#comment-15607792
 ] 

Hive QA commented on HIVE-15056:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12835270/HIVE-15056.2.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1811/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1811/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1811/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-10-26 07:57:24.146
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-1811/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-10-26 07:57:24.148
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 41fbe7b HIVE-13873: Support column pruning for struct fields in 
select statement(Ferdinand Xu via Sun Chao)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 41fbe7b HIVE-13873: Support column pruning for struct fields in 
select statement(Ferdinand Xu via Sun Chao)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-10-26 07:57:25.091
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: patch failed: 
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java:114
error: 
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java: 
patch does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12835270 - PreCommit-HIVE-Build

> Support index shifting for struct fields
> 
>
> Key: HIVE-15056
> URL: https://issues.apache.org/jira/browse/HIVE-15056
> Project: Hive
>  Issue Type: Sub-task
>  Components: File Formats, Physical Optimizer, 
> Serializers/Deserializers
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-15056.1.patch, HIVE-15056.2.patch
>
>
> In HIVE-13873, the following case doesn't work:
> {code}
> select s.c from tbl
> {code}
> where {{tbl}} is of schema:
> {code}
> a int
> s struct
> {code}
> This is because currently we generate a "pruned" schema (in terms of 
> {{GroupType}}) for Parquet reader to scan the data. However, on the Hive side 
> the object inspector still uses the original schema. In particular, in this 
> case for {{s.c}} the data returned by Parquet reader is in index 0, but the 
> object inspector tries to read it in index 1. Therefore, in correct result 
> will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15056) Support index shifting for struct fields

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15607520#comment-15607520
 ] 

Hive QA commented on HIVE-15056:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12835257/HIVE-15056.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10622 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_bulk] 
(batchId=89)
org.apache.hadoop.hive.ql.io.parquet.TestMapStructures.testDoubleMapWithStructValue
 (batchId=246)
org.apache.hadoop.hive.ql.io.parquet.TestMapStructures.testMapWithComplexKey 
(batchId=246)
org.apache.hadoop.hive.ql.io.parquet.TestMapStructures.testNestedMap 
(batchId=246)
org.apache.hadoop.hive.ql.io.parquet.TestMapStructures.testStringMapOfOptionalArray
 (batchId=246)
org.apache.hadoop.hive.ql.io.parquet.TestMapStructures.testStringMapOfOptionalIntArray
 (batchId=246)
org.apache.hadoop.hive.ql.io.parquet.TestMapStructures.testStringMapOptionalPrimitive
 (batchId=246)
org.apache.hadoop.hive.ql.io.parquet.TestMapStructures.testStringMapRequiredPrimitive
 (batchId=246)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
 (batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] (batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] (batchId=164)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1807/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1807/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1807/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12835257 - PreCommit-HIVE-Build

> Support index shifting for struct fields
> 
>
> Key: HIVE-15056
> URL: https://issues.apache.org/jira/browse/HIVE-15056
> Project: Hive
>  Issue Type: Sub-task
>  Components: File Formats, Physical Optimizer, 
> Serializers/Deserializers
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-15056.1.patch
>
>
> In HIVE-13873, the following case doesn't work:
> {code}
> select s.c from tbl
> {code}
> where {{tbl}} is of schema:
> {code}
> a int
> s struct
> {code}
> This is because currently we generate a "pruned" schema (in terms of 
> {{GroupType}}) for Parquet reader to scan the data. However, on the Hive side 
> the object inspector still uses the original schema. In particular, in this 
> case for {{s.c}} the data returned by Parquet reader is in index 0, but the 
> object inspector tries to read it in index 1. Therefore, in correct result 
> will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15056) Support index shifting for struct fields

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15607286#comment-15607286
 ] 

ASF GitHub Bot commented on HIVE-15056:
---

GitHub user sunchao opened a pull request:

https://github.com/apache/hive/pull/108

HIVE-15056: Support index shifting for struct fields



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sunchao/hive column-pruning

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/108.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #108


commit 1532f9e36551eb695fe832d97f31b4dce3eda15b
Author: Chao Sun 
Date:   2016-10-25T21:12:02Z

HIVE-15056: Support index shifting for struct fields




> Support index shifting for struct fields
> 
>
> Key: HIVE-15056
> URL: https://issues.apache.org/jira/browse/HIVE-15056
> Project: Hive
>  Issue Type: Sub-task
>  Components: File Formats, Physical Optimizer, 
> Serializers/Deserializers
>Reporter: Chao Sun
>Assignee: Chao Sun
>
> In HIVE-13873, the following case doesn't work:
> {code}
> select s.c from tbl
> {code}
> where {{tbl}} is of schema:
> {code}
> a int
> s struct
> {code}
> This is because currently we generate a "pruned" schema (in terms of 
> {{GroupType}}) for Parquet reader to scan the data. However, on the Hive side 
> the object inspector still uses the original schema. In particular, in this 
> case for {{s.c}} the data returned by Parquet reader is in index 0, but the 
> object inspector tries to read it in index 1. Therefore, in correct result 
> will be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)