[jira] [Commented] (HIVE-19638) Configuration not passed to ORC Reader.Options
[ https://issues.apache.org/jira/browse/HIVE-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16800545#comment-16800545 ] Mass Dosage commented on HIVE-19638: [~owen.omalley] As discussed, would you be able to help us move this patch forward or put us in touch with some other Hive committer who can? > Configuration not passed to ORC Reader.Options > -- > > Key: HIVE-19638 > URL: https://issues.apache.org/jira/browse/HIVE-19638 > Project: Hive > Issue Type: Bug > Components: Configuration, File Formats, ORC >Affects Versions: 2.3.0, 2.3.1, 2.3.2 >Reporter: Rentao Wu >Assignee: Rentao Wu >Priority: Major > Attachments: HIVE-19638.patch > > > Configuration is not passed to ORC's Reader.Option in OrcFileInputFormat > which causes some [ORC > configurations|https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/Reader.java#L170-L176] > to not be able to be picked up. > Related issues: > For example, the ORC upgrade in Hive 2.3.x changed schema evolution from > positional to column name matching. A backwards compatibility configuration > "orc.force.positional.evolution" could be set in ORC Reader.Options by > [ORC-120|https://issues.apache.org/jira/browse/ORC-120] however it could not > be picked up resulting in null values when querying ORC tables where the > column names do not match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19638) Configuration not passed to ORC Reader.Options
[
https://issues.apache.org/jira/browse/HIVE-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781096#comment-16781096
]
Hive QA commented on HIVE-19638:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924427/HIVE-19638.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15824 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=260)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16299/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16299/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16299/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12924427 - PreCommit-HIVE-Build
> Configuration not passed to ORC Reader.Options
> --
>
> Key: HIVE-19638
> URL: https://issues.apache.org/jira/browse/HIVE-19638
> Project: Hive
> Issue Type: Bug
> Components: Configuration, File Formats, ORC
>Affects Versions: 2.3.0, 2.3.1, 2.3.2
>Reporter: Rentao Wu
>Assignee: Rentao Wu
>Priority: Major
> Attachments: HIVE-19638.patch
>
>
> Configuration is not passed to ORC's Reader.Option in OrcFileInputFormat
> which causes some [ORC
> configurations|https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/Reader.java#L170-L176]
> to not be able to be picked up.
> Related issues:
> For example, the ORC upgrade in Hive 2.3.x changed schema evolution from
> positional to column name matching. A backwards compatibility configuration
> "orc.force.positional.evolution" could be set in ORC Reader.Options by
> [ORC-120|https://issues.apache.org/jira/browse/ORC-120] however it could not
> be picked up resulting in null values when querying ORC tables where the
> column names do not match.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19638) Configuration not passed to ORC Reader.Options
[
https://issues.apache.org/jira/browse/HIVE-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780985#comment-16780985
]
Hive QA commented on HIVE-19638:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 12s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-16299/patches/PreCommit-HIVE-Build-16299.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16299/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Configuration not passed to ORC Reader.Options
> --
>
> Key: HIVE-19638
> URL: https://issues.apache.org/jira/browse/HIVE-19638
> Project: Hive
> Issue Type: Bug
> Components: Configuration, File Formats, ORC
>Affects Versions: 2.3.0, 2.3.1, 2.3.2
>Reporter: Rentao Wu
>Assignee: Rentao Wu
>Priority: Major
> Attachments: HIVE-19638.patch
>
>
> Configuration is not passed to ORC's Reader.Option in OrcFileInputFormat
> which causes some [ORC
> configurations|https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/Reader.java#L170-L176]
> to not be able to be picked up.
> Related issues:
> For example, the ORC upgrade in Hive 2.3.x changed schema evolution from
> positional to column name matching. A backwards compatibility configuration
> "orc.force.positional.evolution" could be set in ORC Reader.Options by
> [ORC-120|https://issues.apache.org/jira/browse/ORC-120] however it could not
> be picked up resulting in null values when querying ORC tables where the
> column names do not match.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19638) Configuration not passed to ORC Reader.Options
[ https://issues.apache.org/jira/browse/HIVE-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780522#comment-16780522 ] Mass Dosage commented on HIVE-19638: OK, so from what I can tell this is about being able to pass a conf variable into the ORC Reader which allows one to override certain ORC properties. I don't think we need an ORC committer to review per se but if you think that's the best path I guess I could ask? In the meantime should I create a properly named patch file for the 2.3.x branch? Could we get the current patch merged into master (assuming it applies cleanly, I haven't checked). > Configuration not passed to ORC Reader.Options > -- > > Key: HIVE-19638 > URL: https://issues.apache.org/jira/browse/HIVE-19638 > Project: Hive > Issue Type: Bug > Components: Configuration, File Formats, ORC >Affects Versions: 2.3.0, 2.3.1, 2.3.2 >Reporter: Rentao Wu >Assignee: Rentao Wu >Priority: Major > Attachments: HIVE-19638.patch > > > Configuration is not passed to ORC's Reader.Option in OrcFileInputFormat > which causes some [ORC > configurations|https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/Reader.java#L170-L176] > to not be able to be picked up. > Related issues: > For example, the ORC upgrade in Hive 2.3.x changed schema evolution from > positional to column name matching. A backwards compatibility configuration > "orc.force.positional.evolution" could be set in ORC Reader.Options by > [ORC-120|https://issues.apache.org/jira/browse/ORC-120] however it could not > be picked up resulting in null values when querying ORC tables where the > column names do not match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19638) Configuration not passed to ORC Reader.Options
[ https://issues.apache.org/jira/browse/HIVE-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780463#comment-16780463 ] Peter Vary commented on HIVE-19638: --- [~massdos...@gmail.com]: This patch has not been merged to any branch yet. As for the ORC related stuff - I am not an expert here. I would look for ORC committers myself: [https://orc.apache.org/news/2016/12/15/new-committers/] Thanks, Peter > Configuration not passed to ORC Reader.Options > -- > > Key: HIVE-19638 > URL: https://issues.apache.org/jira/browse/HIVE-19638 > Project: Hive > Issue Type: Bug > Components: Configuration, File Formats, ORC >Affects Versions: 2.3.0, 2.3.1, 2.3.2 >Reporter: Rentao Wu >Assignee: Rentao Wu >Priority: Major > Attachments: HIVE-19638.patch > > > Configuration is not passed to ORC's Reader.Option in OrcFileInputFormat > which causes some [ORC > configurations|https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/Reader.java#L170-L176] > to not be able to be picked up. > Related issues: > For example, the ORC upgrade in Hive 2.3.x changed schema evolution from > positional to column name matching. A backwards compatibility configuration > "orc.force.positional.evolution" could be set in ORC Reader.Options by > [ORC-120|https://issues.apache.org/jira/browse/ORC-120] however it could not > be picked up resulting in null values when querying ORC tables where the > column names do not match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19638) Configuration not passed to ORC Reader.Options
[ https://issues.apache.org/jira/browse/HIVE-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780457#comment-16780457 ] Mass Dosage commented on HIVE-19638: Has this been merged into any released versions of Hive? We'd really like it in the 2.3.x line and I'd be happy to help with the changes needed to get it in there. [~pvary] is this something you could help with or point me in the direction of a Hive committer who could assist getting this patched? The test failures above to me don't look like they are related to the patch and are rather the usual background noise of Hive test failures. > Configuration not passed to ORC Reader.Options > -- > > Key: HIVE-19638 > URL: https://issues.apache.org/jira/browse/HIVE-19638 > Project: Hive > Issue Type: Bug > Components: Configuration, File Formats, ORC >Affects Versions: 2.3.0, 2.3.1, 2.3.2 >Reporter: Rentao Wu >Assignee: Rentao Wu >Priority: Major > Attachments: HIVE-19638.patch > > > Configuration is not passed to ORC's Reader.Option in OrcFileInputFormat > which causes some [ORC > configurations|https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/Reader.java#L170-L176] > to not be able to be picked up. > Related issues: > For example, the ORC upgrade in Hive 2.3.x changed schema evolution from > positional to column name matching. A backwards compatibility configuration > "orc.force.positional.evolution" could be set in ORC Reader.Options by > [ORC-120|https://issues.apache.org/jira/browse/ORC-120] however it could not > be picked up resulting in null values when querying ORC tables where the > column names do not match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19638) Configuration not passed to ORC Reader.Options
[ https://issues.apache.org/jira/browse/HIVE-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494021#comment-16494021 ] Rentao Wu commented on HIVE-19638: -- Attached reviewboard link > Configuration not passed to ORC Reader.Options > -- > > Key: HIVE-19638 > URL: https://issues.apache.org/jira/browse/HIVE-19638 > Project: Hive > Issue Type: Bug > Components: Configuration, File Formats, ORC >Affects Versions: 2.3.0, 2.3.1, 2.3.2 >Reporter: Rentao Wu >Assignee: Rentao Wu >Priority: Major > Attachments: HIVE-19638.patch > > > Configuration is not passed to ORC's Reader.Option in OrcFileInputFormat > which causes some [ORC > configurations|https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/Reader.java#L170-L176] > to not be able to be picked up. > Related issues: > For example, the ORC upgrade in Hive 2.3.x changed schema evolution from > positional to column name matching. A backwards compatibility configuration > "orc.force.positional.evolution" could be set in ORC Reader.Options by > [ORC-120|https://issues.apache.org/jira/browse/ORC-120] however it could not > be picked up resulting in null values when querying ORC tables where the > column names do not match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19638) Configuration not passed to ORC Reader.Options
[
https://issues.apache.org/jira/browse/HIVE-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484264#comment-16484264
]
Hive QA commented on HIVE-19638:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924427/HIVE-19638.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 14313 tests
executed
*Failed tests:*
{noformat}
TestBeeLineExceptionHandling - did not produce a TEST-*.xml file (likely timed
out) (batchId=192)
TestBeeLineHistory - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
TestBeelineArgParsing - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
TestClientCommandHookFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=192)
TestHiveCli - did not produce a TEST-*.xml file (likely timed out) (batchId=192)
TestHiveSchemaTool - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
TestIncrementalRows - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
TestShutdownHook - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
TestTableOutputFormat - did not produce a TEST-*.xml file (likely timed out)
(batchId=192)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11137/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11137/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11137/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12924427 - PreCommit-HIVE-Build
> Configuration not passed to ORC Reader.Options
> --
>
> Key: HIVE-19638
> URL: https://issues.apache.org/jira/browse/HIVE-19638
> Project: Hive
> Issue Type: Bug
> Components: Configuration, File Formats, ORC
>Affects Versions: 2.3.0, 2.3.1, 2.3.2
>Reporter: Rentao Wu
>Assignee: Rentao Wu
>Priority: Major
> Attachments: HIVE-19638.patch
>
>
> Configuration is not passed to ORC's Reader.Option in OrcFileInputFormat
> which causes some [ORC
> configurations|https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/Reader.java#L170-L176]
> to not be able to be picked up.
> Related issues:
> For example, the ORC upgrade in Hive 2.3.x changed schema evolution from
> positional to column name matching. A backwards compatibility configuration
> "orc.force.positional.evolution" could be set in ORC Reader.Options by
> [ORC-120|https://issues.apache.org/jira/browse/ORC-120] however it could not
> be picked up resulting in null values when querying ORC tables where the
> column names do not match.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19638) Configuration not passed to ORC Reader.Options
[
https://issues.apache.org/jira/browse/HIVE-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16484179#comment-16484179
]
Hive QA commented on HIVE-19638:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 2 new + 341 unchanged - 0
fixed = 343 total (was 341) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11137/dev-support/hive-personality.sh
|
| git revision | master / 7b61c76 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11137/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11137/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Configuration not passed to ORC Reader.Options
> --
>
> Key: HIVE-19638
> URL: https://issues.apache.org/jira/browse/HIVE-19638
> Project: Hive
> Issue Type: Bug
> Components: Configuration, File Formats, ORC
>Affects Versions: 2.3.0, 2.3.1, 2.3.2
>Reporter: Rentao Wu
>Assignee: Rentao Wu
>Priority: Major
> Attachments: HIVE-19638.patch
>
>
> Configuration is not passed to ORC's Reader.Option in OrcFileInputFormat
> which causes some [ORC
> configurations|https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/Reader.java#L170-L176]
> to not be able to be picked up.
> Related issues:
> For example, the ORC upgrade in Hive 2.3.x changed schema evolution from
> positional to column name matching. A backwards compatibility configuration
> "orc.force.positional.evolution" could be set in ORC Reader.Options by
> [ORC-120|https://issues.apache.org/jira/browse/ORC-120] however it could not
> be picked up resulting in null values when querying ORC tables where the
> column names do not match.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19638) Configuration not passed to ORC Reader.Options
[ https://issues.apache.org/jira/browse/HIVE-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483134#comment-16483134 ] Rentao Wu commented on HIVE-19638: -- Attached patch > Configuration not passed to ORC Reader.Options > -- > > Key: HIVE-19638 > URL: https://issues.apache.org/jira/browse/HIVE-19638 > Project: Hive > Issue Type: Bug > Components: Configuration, File Formats, ORC >Affects Versions: 2.3.0, 2.3.1, 2.3.2 >Reporter: Rentao Wu >Assignee: Rentao Wu >Priority: Major > Attachments: HIVE-19638.patch > > > Configuration is not passed to ORC's Reader.Option in OrcFileInputFormat > which causes some [ORC > configurations|https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/Reader.java#L170-L176] > to not be able to be picked up. > Related issues: > For example, the ORC upgrade in Hive 2.3.x changed schema evolution from > positional to column name matching. A backwards compatibility configuration > "orc.force.positional.evolution" could be set in ORC Reader.Options by > [ORC-120|https://issues.apache.org/jira/browse/ORC-120] however it could not > be picked up resulting in null values when querying ORC tables where the > column names do not match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
