[jira] [Commented] (HIVE-19494) Accept shade prefix during reflective instantiation of output format
[
https://issues.apache.org/jira/browse/HIVE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474995#comment-16474995
]
Hive QA commented on HIVE-19494:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923257/HIVE-19494-branch-3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 14207 tests
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=253)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=253)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSSLWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=254)
TestTezPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_vector_dynpart_hashjoin_1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_2]
(batchId=162)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insertsel_fail]
(batchId=95)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=225)
org.apache.hadoop.hive.druid.serde.TestDruidSerDe.testDruidObjectDeserializer
(batchId=261)
org.apache.hadoop.hive.druid.serde.TestDruidSerDe.testDruidObjectSerializer
(batchId=261)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testApplyPlanQpChanges
(batchId=293)
org.apache.hive.service.server.TestInformationSchemaWithPrivilege.test
(batchId=238)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10948/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10948/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10948/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923257 - PreCommit-HIVE-Build
> Accept shade prefix during reflective instantiation of output format
>
>
> Key: HIVE-19494
> URL: https://issues.apache.org/jira/browse/HIVE-19494
> Project: Hive
> Issue Type: Sub-task
> Components: Streaming
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19494-branch-3.patch, HIVE-19494.1.patch
>
>
> Hive Streaming API jars are sometimes shaded with a different prefix when
> used in environments where another version of hive already exists (spark for
> example). In most cases, shading is done with rename of classes with some
> prefix. If an uber/assembly jar is generated with renamed prefix, Hive
> Streaming API will not work as Hive Streaming API will reflectively
> instantiate outputformat class using FQCN string provided by metastore table
> storage descriptor object.
> For example:
> RecordWriter will create instance of OutputFormat using string
> "org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat". When a shaded jar with
> renamed class references are used, this class will not be found by the
> classloader.
> We can optionally accept a shade prefix from user via config which will be
> tried (as fallback) when ClassNotFoundException is thrown.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19494) Accept shade prefix during reflective instantiation of output format
[
https://issues.apache.org/jira/browse/HIVE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474838#comment-16474838
]
Hive QA commented on HIVE-19494:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10948/dev-support/hive-personality.sh
|
| git revision | master / 042eb39 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common streaming U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10948/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Accept shade prefix during reflective instantiation of output format
>
>
> Key: HIVE-19494
> URL: https://issues.apache.org/jira/browse/HIVE-19494
> Project: Hive
> Issue Type: Sub-task
> Components: Streaming
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19494-branch-3.patch, HIVE-19494.1.patch
>
>
> Hive Streaming API jars are sometimes shaded with a different prefix when
> used in environments where another version of hive already exists (spark for
> example). In most cases, shading is done with rename of classes with some
> prefix. If an uber/assembly jar is generated with renamed prefix, Hive
> Streaming API will not work as Hive Streaming API will reflectively
> instantiate outputformat class using FQCN string provided by metastore table
> storage descriptor object.
> For example:
> RecordWriter will create instance of OutputFormat using string
> "org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat". When a shaded jar with
> renamed class references are
[jira] [Commented] (HIVE-19494) Accept shade prefix during reflective instantiation of output format
[
https://issues.apache.org/jira/browse/HIVE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472973#comment-16472973
]
Hive QA commented on HIVE-19494:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12922922/HIVE-19494.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 36 failed/errored test(s), 13578 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Commented] (HIVE-19494) Accept shade prefix during reflective instantiation of output format
[
https://issues.apache.org/jira/browse/HIVE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472955#comment-16472955
]
Hive QA commented on HIVE-19494:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10847/dev-support/hive-personality.sh
|
| git revision | master / 615e71e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common streaming U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10847/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Accept shade prefix during reflective instantiation of output format
>
>
> Key: HIVE-19494
> URL: https://issues.apache.org/jira/browse/HIVE-19494
> Project: Hive
> Issue Type: Sub-task
> Components: Streaming
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19494.1.patch
>
>
> Hive Streaming API jars are sometimes shaded with a different prefix when
> used in environments where another version of hive already exists (spark for
> example). In most cases, shading is done with rename of classes with some
> prefix. If an uber/assembly jar is generated with renamed prefix, Hive
> Streaming API will not work as Hive Streaming API will reflectively
> instantiate outputformat class using FQCN string provided by metastore table
> storage descriptor object.
> For example:
> RecordWriter will create instance of OutputFormat using string
> "org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat". When a shaded jar with
> renamed class references are used, this class will not be
[jira] [Commented] (HIVE-19494) Accept shade prefix during reflective instantiation of output format
[ https://issues.apache.org/jira/browse/HIVE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471216#comment-16471216 ] Prasanth Jayachandran commented on HIVE-19494: -- Only streaming ingest encountered this recently, but technically this can happen for the case where multiple hive versions are involved. Spark compiles against 1.2.1 but streaming ingest API requires hive 3.0.0 and in its codepath, so only this instantiation in hive streaming API fails as of now (in my limited testing). I left the config generic enough for future in case if someone runs into the same issue they can reuse the hive config. > Accept shade prefix during reflective instantiation of output format > > > Key: HIVE-19494 > URL: https://issues.apache.org/jira/browse/HIVE-19494 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19494.1.patch > > > Hive Streaming API jars are sometimes shaded with a different prefix when > used in environments where another version of hive already exists (spark for > example). In most cases, shading is done with rename of classes with some > prefix. If an uber/assembly jar is generated with renamed prefix, Hive > Streaming API will not work as Hive Streaming API will reflectively > instantiate outputformat class using FQCN string provided by metastore table > storage descriptor object. > For example: > RecordWriter will create instance of OutputFormat using string > "org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat". When a shaded jar with > renamed class references are used, this class will not be found by the > classloader. > We can optionally accept a shade prefix from user via config which will be > tried (as fallback) when ClassNotFoundException is thrown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19494) Accept shade prefix during reflective instantiation of output format
[ https://issues.apache.org/jira/browse/HIVE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471200#comment-16471200 ] Sergey Shelukhin commented on HIVE-19494: - The name and description seem to indicate this is a global setting for everywhere Hive does this, however it's only used in streaming. Should it be scoped to streaming w.r.t. name and desc? Otherwise +1 > Accept shade prefix during reflective instantiation of output format > > > Key: HIVE-19494 > URL: https://issues.apache.org/jira/browse/HIVE-19494 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19494.1.patch > > > Hive Streaming API jars are sometimes shaded with a different prefix when > used in environments where another version of hive already exists (spark for > example). In most cases, shading is done with rename of classes with some > prefix. If an uber/assembly jar is generated with renamed prefix, Hive > Streaming API will not work as Hive Streaming API will reflectively > instantiate outputformat class using FQCN string provided by metastore table > storage descriptor object. > For example: > RecordWriter will create instance of OutputFormat using string > "org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat". When a shaded jar with > renamed class references are used, this class will not be found by the > classloader. > We can optionally accept a shade prefix from user via config which will be > tried (as fallback) when ClassNotFoundException is thrown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19494) Accept shade prefix during reflective instantiation of output format
[ https://issues.apache.org/jira/browse/HIVE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471192#comment-16471192 ] Prasanth Jayachandran commented on HIVE-19494: -- [~sershe] can you please take a look? small patch that accepts a prefix for classloading. > Accept shade prefix during reflective instantiation of output format > > > Key: HIVE-19494 > URL: https://issues.apache.org/jira/browse/HIVE-19494 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19494.1.patch > > > Hive Streaming API jars are sometimes shaded with a different prefix when > used in environments where another version of hive already exists (spark for > example). In most cases, shading is done with rename of classes with some > prefix. If an uber/assembly jar is generated with renamed prefix, Hive > Streaming API will not work as Hive Streaming API will reflectively > instantiate outputformat class using FQCN string provided by metastore table > storage descriptor object. > For example: > RecordWriter will create instance of OutputFormat using string > "org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat". When a shaded jar with > renamed class references are used, this class will not be found by the > classloader. > We can optionally accept a shade prefix from user via config which will be > tried (as fallback) when ClassNotFoundException is thrown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
