[jira] [Commented] (TEZ-4630) Issue related to Tez-0.10.5 rc0
[
https://issues.apache.org/jira/browse/TEZ-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17951989#comment-17951989
]
Butao Zhang commented on TEZ-4630:
--
I tried to discover the differences in deploying Tez between you and me. :)
# The Tez binary package i used is
[https://dist.apache.org/repos/dist/dev/tez/apache-tez-0.10.5-rc0/apache-tez-0.10.5-bin.tar.gz]
, which contains several hadoop 3.4.1 jars in lib directory,
and{color:#de350b} these 3.4.1 jars could cause Beeline failed {color}with
java.lang.NoClassDefFoundError: Could not initialize class
com.sun.proxy.$Proxy29 (state=08S01,code=0) Could you double check if you also
used this binary package?
{code:java}
[root@locahost data]# ll apache-tez-0.10.5-bin/lib/hadoop-*
-rwxrwxr-x 1 501 games 5.3M Jan 22 2020
apache-tez-0.10.5-bin/lib/hadoop-hdfs-client-3.4.1.jar
-rwxrwxr-x 1 501 games 772K Jan 22 2020
apache-tez-0.10.5-bin/lib/hadoop-mapreduce-client-common-3.4.1.jar
-rwxrwxr-x 1 501 games 1.8M Jan 22 2020
apache-tez-0.10.5-bin/lib/hadoop-mapreduce-client-core-3.4.1.jar
-rwxrwxr-x 1 501 games 61K Jan 22 2020
apache-tez-0.10.5-bin/lib/hadoop-yarn-server-timeline-pluginstorage-3.4.1.jar
{code}
# The hive binary package i used is latest hive master branch, and i did not
change its Tez version to 0.10.5(Iam not using
[https://github.com/apache/hive/pull/5802|https://github.com/apache/hive/pull/5802).]).
BTW, i found the hive binary package also includes tez
jars({color:#de350b}tez-api-0.10.4.jar{color}), so this may also cause some
issues if i want try to submit tez job against tez-0.10.5 package.
{color:#de350b}IMO, Hadoop 3.3.6 & Tez0.10.5 & Hive lastest branch can work
together, but there are some prerequisites:{color}
* Users should change hadoop dependency to 3.3.6 & tez dependency to 0.10.5 in
hive pom.xml, and re-build hive to get a correct hive binary package.
* Users should change hadoop dependency to 3.3.6 in tez pom.xml, and re-build
tez source code to get a correct tez binary package.
> Issue related to Tez-0.10.5 rc0
> ---
>
> Key: TEZ-4630
> URL: https://issues.apache.org/jira/browse/TEZ-4630
> Project: Apache Tez
> Issue Type: Test
>Reporter: Butao Zhang
>Priority: Major
> Attachments: tez-hive-hadoop-3.3.6.rtf
>
>
> Test Env: *Hadoop3.3.6* & Hive latest master branch, and Tez 0.10.5
> [https://dist.apache.org/repos/dist/dev/tez/apache-tez-0.10.5-rc0/apache-tez-0.10.5-bin.tar.gz]
>
> *NOTE:* if using {color:#de350b}*Hadoop3.4.1, the issue will disappear.*
> {color}
>
> Found that Beeline session will fail with:
> {color:#FF}java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29{color}
> {code:java}
> zbt@zbt-ubuntu:~$ /usr/tools/apache-hive-beeline-4.1.0-SNAPSHOT/bin/beeline
> -u "jdbc:hive2://127.0.0.1:1/default hive"HADOOP_HOME not set, executing
> beeline using JAVAConnecting to jdbc:hive2://127.0.0.1:1/defaultError:
> Could not open client transport with JDBC Uri:
> jdbc:hive2://100.71.9.13:10004/default: Failed to open new session:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29 (state=08S01,code=0) {code}
> {color:#FF}Detailed stack trace in HS2 log:{color}
> {code:java}
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Client protocol version:
> HIVE_CLI_SERVICE_PROTOCOL_V10
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Creating Hive session handle for user [null] from IP
> 100.72.1.9
> 2025-05-15T09:59:09,995 WARN [HiveServer2-Handler-Pool: Thread-74]
> service.CompositeService: Failed to open session
> java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could not
> initialize class com.sun.proxy.$Proxy29
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_221]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_221]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> ~[hadoop-common-3.3.6.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at com.sun.proxy.$Proxy40.open(Unknown Source) ~[?:?]
> at
>
[jira] [Commented] (TEZ-4630) Issue related to Tez-0.10.5 rc0
[
https://issues.apache.org/jira/browse/TEZ-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17951820#comment-17951820
]
Ayush Saxena commented on TEZ-4630:
---
I deployed Hadoop-3.3.6 locally. Build Hive master with Laszlo's Tez Upgrade PR
in Hive ([https://github.com/apache/hive/pull/5802).] And with derby DB. Tried
both Tez local mode & non-local mode & it was success.
Hive does bring in hadoop-3.4.1 in classpath because they are there in the lib
directory, but it didn't create problem for me & the query succeeded.
My steps:
I had HADOOP_HOME, TEZ_HOME, TEZ_CONF_DIR, TEZ_JARS
{noformat}
export HADOOP_HOME=/Users/ayushsaxena/code/incubator/hadoop/hadoop-3.3.6
export
TEZ_HOME=/Users/ayushsaxena/code/incubator/tez/apache-tez-0.10.5-src/tez-dist/target/tez-0.10.5
export TEZ_CONF_DIR=$TEZ_HOME/conf
export
TEZ_JARS=/Users/ayushsaxena/code/incubator/tez/apache-tez-0.10.5-src/tez-dist/target/tez-0.10.5
{noformat}
My HADOOP_CLASSPATH before starting HS2 & beeline
{noformat}
export
HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:${HADOOP_CLASSPATH}:
${JAVA_JDBC_LIBS}:${MAPREDUCE_LIBS}{noformat}
Then I did had these libs stored in HDFS
{noformat}
$HADOOP_HOME/bin/hdfs dfs -mkdir -p hdfs://localhost:52714/apps/tez/
$HADOOP_HOME/bin/hdfs dfs -copyFromLocal ../tez-0.10.5.tar.gz
hdfs://localhost:52714/apps/tez/
$HADOOP_HOME/bin/hdfs dfs -copyFromLocal ../tez-0.10.5/
hdfs://localhost:52714/apps/tez/ {noformat}
My Tez-site.xml
{noformat}
tez.lib.uris
hdfs://localhost:52714/apps/tez/tez-0.10.5.tar.gz,hdfs://localhost:52714/apps/tez/tez-0.10.5/lib,hdfs://localhost:52714/apps/tez/tez-0.10.5
tez.use.cluster.hadoop-libs
true
{noformat}
My Hive-site.xml had some extra stuff since before, I believe this is Tez
related.
{noformat}
hive.support.concurrency
true
tez.lib.uris
hdfs://localhost:52714/apps/tez/tez-0.10.5.tar.gz,hdfs://localhost:52714/apps/tez/tez-0.10.5/lib,hdfs://localhost:52714/apps/tez/tez-0.10.5
tez.user.classpath.first
true
Boolean value.
Specify whether the user classpath takes precedence over the Tez framework
classpath.
boolean
tez.use.cluster.hadoop-libs
true
boolean
{noformat}
Bunch of things might not be required, it is all my old steps, which I didn't
cleanup, but still Hive-On-Tez worked for me with 0.10.5 & Hadoop-3.3.6 & Hive
compiled with Laszlo's Tez Upgrade PR.
Let me know if I missed something, attaching my beeline query log as well
> Issue related to Tez-0.10.5 rc0
> ---
>
> Key: TEZ-4630
> URL: https://issues.apache.org/jira/browse/TEZ-4630
> Project: Apache Tez
> Issue Type: Test
>Reporter: Butao Zhang
>Priority: Major
> Attachments: tez-hive-hadoop-3.3.6.rtf
>
>
> Test Env: *Hadoop3.3.6* & Hive latest master branch, and Tez 0.10.5
> [https://dist.apache.org/repos/dist/dev/tez/apache-tez-0.10.5-rc0/apache-tez-0.10.5-bin.tar.gz]
>
> *NOTE:* if using {color:#de350b}*Hadoop3.4.1, the issue will disappear.*
> {color}
>
> Found that Beeline session will fail with:
> {color:#FF}java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29{color}
> {code:java}
> zbt@zbt-ubuntu:~$ /usr/tools/apache-hive-beeline-4.1.0-SNAPSHOT/bin/beeline
> -u "jdbc:hive2://127.0.0.1:1/default hive"HADOOP_HOME not set, executing
> beeline using JAVAConnecting to jdbc:hive2://127.0.0.1:1/defaultError:
> Could not open client transport with JDBC Uri:
> jdbc:hive2://100.71.9.13:10004/default: Failed to open new session:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29 (state=08S01,code=0) {code}
> {color:#FF}Detailed stack trace in HS2 log:{color}
> {code:java}
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Client protocol version:
> HIVE_CLI_SERVICE_PROTOCOL_V10
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Creating Hive session handle for user [null] from IP
> 100.72.1.9
> 2025-05-15T09:59:09,995 WARN [HiveServer2-Handler-Pool: Thread-74]
> service.CompositeService: Failed to open session
> java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could not
> initialize class com.sun.proxy.$Proxy29
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
>
[jira] [Commented] (TEZ-4630) Issue related to Tez-0.10.5 rc0
[
https://issues.apache.org/jira/browse/TEZ-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17951680#comment-17951680
]
Butao Zhang commented on TEZ-4630:
--
IMO, I believe that if users want to use hadoop 3.3.6 & Tez0.10.5,
{color:#de350b}They'd better recompile hive&Tez0.10.5 against Hadoop3.3.6 to
regenerate a tez binary tar. This can eliminate all kinds of strange problems
caused by the version of Hadoop.{color}
Therefore, I believe there is not much necessity for us to study this issue.
Users need to figure out for themselves the various problems that different
Hadoop versions may cause.
BTW, we might consider releasing binary packages that do not include the hadoop
jar, just like binary package in
[https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-3.5.5/] like
[spark-3.5.5-bin-without-hadoop.tgz
|https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-3.5.5/spark-3.5.5-bin-without-hadoop.tgz]and
[spark-3.5.5-bin-hadoop3.tgz
|https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz]
This may can help to avoid as many issues as possible that may arise from the
inconsistency between the Tez's built-in Hadoop package and the local hadoop
deployment version.
> Issue related to Tez-0.10.5 rc0
> ---
>
> Key: TEZ-4630
> URL: https://issues.apache.org/jira/browse/TEZ-4630
> Project: Apache Tez
> Issue Type: Test
>Reporter: Butao Zhang
>Priority: Major
>
> Test Env: *Hadoop3.3.6* & Hive latest master branch, and Tez 0.10.5
> [https://dist.apache.org/repos/dist/dev/tez/apache-tez-0.10.5-rc0/apache-tez-0.10.5-bin.tar.gz]
>
> *NOTE:* if using {color:#de350b}*Hadoop3.4.1, the issue will disappear.*
> {color}
>
> Found that Beeline session will fail with:
> {color:#FF}java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29{color}
> {code:java}
> zbt@zbt-ubuntu:~$ /usr/tools/apache-hive-beeline-4.1.0-SNAPSHOT/bin/beeline
> -u "jdbc:hive2://127.0.0.1:1/default hive"HADOOP_HOME not set, executing
> beeline using JAVAConnecting to jdbc:hive2://127.0.0.1:1/defaultError:
> Could not open client transport with JDBC Uri:
> jdbc:hive2://100.71.9.13:10004/default: Failed to open new session:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29 (state=08S01,code=0) {code}
> {color:#FF}Detailed stack trace in HS2 log:{color}
> {code:java}
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Client protocol version:
> HIVE_CLI_SERVICE_PROTOCOL_V10
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Creating Hive session handle for user [null] from IP
> 100.72.1.9
> 2025-05-15T09:59:09,995 WARN [HiveServer2-Handler-Pool: Thread-74]
> service.CompositeService: Failed to open session
> java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could not
> initialize class com.sun.proxy.$Proxy29
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_221]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_221]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> ~[hadoop-common-3.3.6.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at com.sun.proxy.$Proxy40.open(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:485)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:425)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:198)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:562)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:406)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thr
[jira] [Commented] (TEZ-4630) Issue related to Tez-0.10.5 rc0
[
https://issues.apache.org/jira/browse/TEZ-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17951673#comment-17951673
]
László Bodor commented on TEZ-4630:
---
thanks for the details! I think we can ignore the original error, as tez.tar.gz
always comes with the corresponding hadoop version to serve as a standalone
package, and its full content is not supposed to be included on other
components' classpath, like: HS2 (which basically supposed to see tez-api only)
regarding DAGAppMaster failure, my question is kinda the same as what I had
regarding the original problem, like: it would be crucial to understand how the
HS2 was started
I cannot repro anything with a MiniHS2, but that's quite different from your
use case:
{code}
mvn install -Dtest=StartMiniHS2Cluster -DminiHS2.clusterType=llap
-DminiHS2.conf="target/testconf/llap/hive-site.xml" -DminiHS2.run=true
-DminiHS2.usePortsFromConf=true -pl itests/hive-unit -Pitests
{code}
and this step becomes much more important when it comes to the DAGAppMaster
failure, because we need to understand how your hs2 started and interacts with
your hadoop installation (my minihs2 creates a minicluster on the fly I believe)
> Issue related to Tez-0.10.5 rc0
> ---
>
> Key: TEZ-4630
> URL: https://issues.apache.org/jira/browse/TEZ-4630
> Project: Apache Tez
> Issue Type: Test
>Reporter: Butao Zhang
>Priority: Major
>
> Test Env: *Hadoop3.3.6* & Hive latest master branch, and Tez 0.10.5
> [https://dist.apache.org/repos/dist/dev/tez/apache-tez-0.10.5-rc0/apache-tez-0.10.5-bin.tar.gz]
>
> *NOTE:* if using {color:#de350b}*Hadoop3.4.1, the issue will disappear.*
> {color}
>
> Found that Beeline session will fail with:
> {color:#FF}java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29{color}
> {code:java}
> zbt@zbt-ubuntu:~$ /usr/tools/apache-hive-beeline-4.1.0-SNAPSHOT/bin/beeline
> -u "jdbc:hive2://127.0.0.1:1/default hive"HADOOP_HOME not set, executing
> beeline using JAVAConnecting to jdbc:hive2://127.0.0.1:1/defaultError:
> Could not open client transport with JDBC Uri:
> jdbc:hive2://100.71.9.13:10004/default: Failed to open new session:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29 (state=08S01,code=0) {code}
> {color:#FF}Detailed stack trace in HS2 log:{color}
> {code:java}
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Client protocol version:
> HIVE_CLI_SERVICE_PROTOCOL_V10
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Creating Hive session handle for user [null] from IP
> 100.72.1.9
> 2025-05-15T09:59:09,995 WARN [HiveServer2-Handler-Pool: Thread-74]
> service.CompositeService: Failed to open session
> java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could not
> initialize class com.sun.proxy.$Proxy29
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_221]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_221]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> ~[hadoop-common-3.3.6.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at com.sun.proxy.$Proxy40.open(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:485)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:425)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:198)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:562)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:406)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLISe
[jira] [Commented] (TEZ-4630) Issue related to Tez-0.10.5 rc0
[
https://issues.apache.org/jira/browse/TEZ-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17951671#comment-17951671
]
László Bodor commented on TEZ-4630:
---
hm, it would be crucial to understand how the HS2 was started, as the exception
is hit on HS2 codepaths
I also deployed hadoop 3.3.6 with pseudo-local mode, and ran HS2 from a branch
which contains the latest hive master:
{code}
mvn install -Dtest=StartMiniHS2Cluster -DminiHS2.clusterType=llap
-DminiHS2.conf="target/testconf/llap/hive-site.xml" -DminiHS2.run=true
-DminiHS2.usePortsFromConf=true -pl itests/hive-unit -Pitests
{code}
but latest hive master depends on hadoop.version=3.4.1, so I guess the mini
cluster that's deployed under MiniHS2 depends on 3.4.1, hence I didn't hit
anything...I'm just wondering what it looks like on your machine that you
installed hadoop 3.3.6 and you somehow "ran HS2 against that"
> Issue related to Tez-0.10.5 rc0
> ---
>
> Key: TEZ-4630
> URL: https://issues.apache.org/jira/browse/TEZ-4630
> Project: Apache Tez
> Issue Type: Test
>Reporter: Butao Zhang
>Priority: Major
>
> Test Env: *Hadoop3.3.6* & Hive latest master branch, and Tez 0.10.5
> [https://dist.apache.org/repos/dist/dev/tez/apache-tez-0.10.5-rc0/apache-tez-0.10.5-bin.tar.gz]
>
> *NOTE:* if using {color:#de350b}*Hadoop3.4.1, the issue will disappear.*
> {color}
>
> Found that Beeline session will fail with:
> {color:#FF}java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29{color}
> {code:java}
> zbt@zbt-ubuntu:~$ /usr/tools/apache-hive-beeline-4.1.0-SNAPSHOT/bin/beeline
> -u "jdbc:hive2://127.0.0.1:1/default hive"HADOOP_HOME not set, executing
> beeline using JAVAConnecting to jdbc:hive2://127.0.0.1:1/defaultError:
> Could not open client transport with JDBC Uri:
> jdbc:hive2://100.71.9.13:10004/default: Failed to open new session:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29 (state=08S01,code=0) {code}
> {color:#FF}Detailed stack trace in HS2 log:{color}
> {code:java}
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Client protocol version:
> HIVE_CLI_SERVICE_PROTOCOL_V10
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Creating Hive session handle for user [null] from IP
> 100.72.1.9
> 2025-05-15T09:59:09,995 WARN [HiveServer2-Handler-Pool: Thread-74]
> service.CompositeService: Failed to open session
> java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could not
> initialize class com.sun.proxy.$Proxy29
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_221]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_221]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> ~[hadoop-common-3.3.6.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at com.sun.proxy.$Proxy40.open(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:485)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:425)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:198)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:562)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:406)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1595)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1575)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(Process
[jira] [Commented] (TEZ-4630) Issue related to Tez-0.10.5 rc0
[
https://issues.apache.org/jira/browse/TEZ-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17951654#comment-17951654
]
Butao Zhang commented on TEZ-4630:
--
I found the root cause. This is because the binary package of Tez0.10.5
includes several jars of Haoop 3.4.1. Then the hadoop 3.4.1 jars got mixed up
with the hadoop 3.3.6 jars, which caused the problem.
ll apache-tez-0.10.5-bin/lib/hadoop-*
{code:java}
-rwxrwxr-x 1 501 games 5.3M Jan 22 2020 hadoop-hdfs-client-3.4.1.jar
-rwxrwxr-x 1 501 games 772K Jan 22 2020
hadoop-mapreduce-client-common-3.4.1.jar
-rwxrwxr-x 1 501 games 1.8M Jan 22 2020 hadoop-mapreduce-client-core-3.4.1.jar
-rwxrwxr-x 1 501 games 61K Jan 22 2020
hadoop-yarn-server-timeline-pluginstorage-3.4.1.jar {code}
After i remove the hadoop jars from tez 0.10.5 binary package, and restart
hadoop&hs2, this issue(Could not initialize class com.sun.proxy.$Proxy29) can
disappear.
But if i submit a hive tez job, i found the tez am failed to start:
{code:java}
2025-05-15 16:44:16,224 [INFO] [main] |common.TezClassLoader|: Created
TezClassLoader with parent classloader:
sun.misc.Launcher$AppClassLoader@6ad5c04e, thread: 1, system classloader:
sun.misc.Launcher$AppClassLoader@6ad5c04e
2025-05-15 16:44:16,406 [ERROR] [main] |app.DAGAppMaster|: Error starting
DAGAppMaster
java.lang.NoSuchMethodError:
org.apache.commons.cli.Option.builder()Lorg/apache/commons/cli/Option$Builder;
at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2404)
{code}
> Issue related to Tez-0.10.5 rc0
> ---
>
> Key: TEZ-4630
> URL: https://issues.apache.org/jira/browse/TEZ-4630
> Project: Apache Tez
> Issue Type: Test
>Reporter: Butao Zhang
>Priority: Major
>
> Test Env: *Hadoop3.3.6* & Hive latest master branch, and Tez 0.10.5
> [https://dist.apache.org/repos/dist/dev/tez/apache-tez-0.10.5-rc0/apache-tez-0.10.5-bin.tar.gz]
>
> *NOTE:* if using {color:#de350b}*Hadoop3.4.1, the issue will disappear.*
> {color}
>
> Found that Beeline session will fail with:
> {color:#FF}java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29{color}
> {code:java}
> zbt@zbt-ubuntu:~$ /usr/tools/apache-hive-beeline-4.1.0-SNAPSHOT/bin/beeline
> -u "jdbc:hive2://127.0.0.1:1/default hive"HADOOP_HOME not set, executing
> beeline using JAVAConnecting to jdbc:hive2://127.0.0.1:1/defaultError:
> Could not open client transport with JDBC Uri:
> jdbc:hive2://100.71.9.13:10004/default: Failed to open new session:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29 (state=08S01,code=0) {code}
> {color:#FF}Detailed stack trace in HS2 log:{color}
> {code:java}
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Client protocol version:
> HIVE_CLI_SERVICE_PROTOCOL_V10
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Creating Hive session handle for user [null] from IP
> 100.72.1.9
> 2025-05-15T09:59:09,995 WARN [HiveServer2-Handler-Pool: Thread-74]
> service.CompositeService: Failed to open session
> java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could not
> initialize class com.sun.proxy.$Proxy29
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_221]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_221]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> ~[hadoop-common-3.3.6.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at com.sun.proxy.$Proxy40.open(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:485)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:425)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:198)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService
[jira] [Commented] (TEZ-4630) Issue related to Tez-0.10.5 rc0
[
https://issues.apache.org/jira/browse/TEZ-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17951614#comment-17951614
]
László Bodor commented on TEZ-4630:
---
thanks [~zhangbutao]! can you share the detailed repro steps regarding: how hs2
was started? how hadoop 3.3.6 was used? (it was present in hs2 or while running
the beeline executable, or both?)
> Issue related to Tez-0.10.5 rc0
> ---
>
> Key: TEZ-4630
> URL: https://issues.apache.org/jira/browse/TEZ-4630
> Project: Apache Tez
> Issue Type: Test
>Reporter: Butao Zhang
>Priority: Major
>
> Test Env: *Hadoop3.3.6* & Hive latest master branch, and Tez 0.10.5
> [https://dist.apache.org/repos/dist/dev/tez/apache-tez-0.10.5-rc0/apache-tez-0.10.5-bin.tar.gz]
>
> *NOTE:* if using {color:#de350b}*Hadoop3.4.1, the issue will disappear.*
> {color}
>
> Found that Beeline session will fail with:
> {color:#FF}java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29{color}
> {code:java}
> zbt@zbt-ubuntu:~$ /usr/tools/apache-hive-beeline-4.1.0-SNAPSHOT/bin/beeline
> -u "jdbc:hive2://127.0.0.1:1/default hive"HADOOP_HOME not set, executing
> beeline using JAVAConnecting to jdbc:hive2://127.0.0.1:1/defaultError:
> Could not open client transport with JDBC Uri:
> jdbc:hive2://100.71.9.13:10004/default: Failed to open new session:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29 (state=08S01,code=0) {code}
> {color:#FF}Detailed stack trace in HS2 log:{color}
> {code:java}
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Client protocol version:
> HIVE_CLI_SERVICE_PROTOCOL_V10
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Creating Hive session handle for user [null] from IP
> 100.72.1.9
> 2025-05-15T09:59:09,995 WARN [HiveServer2-Handler-Pool: Thread-74]
> service.CompositeService: Failed to open session
> java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could not
> initialize class com.sun.proxy.$Proxy29
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_221]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_221]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> ~[hadoop-common-3.3.6.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at com.sun.proxy.$Proxy40.open(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:485)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:425)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:198)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:562)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:406)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1595)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1575)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> java.ut
[jira] [Commented] (TEZ-4630) Issue related to Tez-0.10.5 rc0
[
https://issues.apache.org/jira/browse/TEZ-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17951636#comment-17951636
]
Butao Zhang commented on TEZ-4630:
--
I compiled hive code using this jdk:
{code:java}
zbt@zbt-ubuntu:~$ java -version
java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode){code}
And I also used this JDK when running Hive&Hadoop.
> Issue related to Tez-0.10.5 rc0
> ---
>
> Key: TEZ-4630
> URL: https://issues.apache.org/jira/browse/TEZ-4630
> Project: Apache Tez
> Issue Type: Test
>Reporter: Butao Zhang
>Priority: Major
>
> Test Env: *Hadoop3.3.6* & Hive latest master branch, and Tez 0.10.5
> [https://dist.apache.org/repos/dist/dev/tez/apache-tez-0.10.5-rc0/apache-tez-0.10.5-bin.tar.gz]
>
> *NOTE:* if using {color:#de350b}*Hadoop3.4.1, the issue will disappear.*
> {color}
>
> Found that Beeline session will fail with:
> {color:#FF}java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29{color}
> {code:java}
> zbt@zbt-ubuntu:~$ /usr/tools/apache-hive-beeline-4.1.0-SNAPSHOT/bin/beeline
> -u "jdbc:hive2://127.0.0.1:1/default hive"HADOOP_HOME not set, executing
> beeline using JAVAConnecting to jdbc:hive2://127.0.0.1:1/defaultError:
> Could not open client transport with JDBC Uri:
> jdbc:hive2://100.71.9.13:10004/default: Failed to open new session:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29 (state=08S01,code=0) {code}
> {color:#FF}Detailed stack trace in HS2 log:{color}
> {code:java}
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Client protocol version:
> HIVE_CLI_SERVICE_PROTOCOL_V10
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Creating Hive session handle for user [null] from IP
> 100.72.1.9
> 2025-05-15T09:59:09,995 WARN [HiveServer2-Handler-Pool: Thread-74]
> service.CompositeService: Failed to open session
> java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could not
> initialize class com.sun.proxy.$Proxy29
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_221]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_221]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> ~[hadoop-common-3.3.6.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at com.sun.proxy.$Proxy40.open(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:485)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:425)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:198)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:562)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:406)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1595)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1575)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolS
[jira] [Commented] (TEZ-4630) Issue related to Tez-0.10.5 rc0
[
https://issues.apache.org/jira/browse/TEZ-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17951634#comment-17951634
]
Stamatis Zampetakis commented on TEZ-4630:
--
This is a strange error since com.sun.proxy.$Proxy29 is a dynamically generated
class. I am suspecting that the error may have something to do with the JDK
that was used to compile the jars vs. the JDK that is used to run them.
> Issue related to Tez-0.10.5 rc0
> ---
>
> Key: TEZ-4630
> URL: https://issues.apache.org/jira/browse/TEZ-4630
> Project: Apache Tez
> Issue Type: Test
>Reporter: Butao Zhang
>Priority: Major
>
> Test Env: *Hadoop3.3.6* & Hive latest master branch, and Tez 0.10.5
> [https://dist.apache.org/repos/dist/dev/tez/apache-tez-0.10.5-rc0/apache-tez-0.10.5-bin.tar.gz]
>
> *NOTE:* if using {color:#de350b}*Hadoop3.4.1, the issue will disappear.*
> {color}
>
> Found that Beeline session will fail with:
> {color:#FF}java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29{color}
> {code:java}
> zbt@zbt-ubuntu:~$ /usr/tools/apache-hive-beeline-4.1.0-SNAPSHOT/bin/beeline
> -u "jdbc:hive2://127.0.0.1:1/default hive"HADOOP_HOME not set, executing
> beeline using JAVAConnecting to jdbc:hive2://127.0.0.1:1/defaultError:
> Could not open client transport with JDBC Uri:
> jdbc:hive2://100.71.9.13:10004/default: Failed to open new session:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29 (state=08S01,code=0) {code}
> {color:#FF}Detailed stack trace in HS2 log:{color}
> {code:java}
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Client protocol version:
> HIVE_CLI_SERVICE_PROTOCOL_V10
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Creating Hive session handle for user [null] from IP
> 100.72.1.9
> 2025-05-15T09:59:09,995 WARN [HiveServer2-Handler-Pool: Thread-74]
> service.CompositeService: Failed to open session
> java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could not
> initialize class com.sun.proxy.$Proxy29
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_221]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_221]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> ~[hadoop-common-3.3.6.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at com.sun.proxy.$Proxy40.open(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:485)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:425)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:198)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:562)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:406)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1595)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1575)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
> ~[hive-exec-4.1.0-SNAP
[jira] [Commented] (TEZ-4630) Issue related to Tez-0.10.5 rc0
[
https://issues.apache.org/jira/browse/TEZ-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17951618#comment-17951618
]
Butao Zhang commented on TEZ-4630:
--
* 1: I added Tez lib into HADOOP_CLASSPATH by changing hadoop configuration
file {color:#de350b}hadoop-env.sh{color} :
{code:java}
if [ -d "/data01/data/apache-tez-0.10.5-bin" ]; then
export
HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/data01/data/apache-tez-0.10.5-bin/*:/data01/data/apache-tez-0.10.5-bin/lib/*:/data01/data/apache-tez-0.10.5-bin/conf
fi
{code}
* 2: Restart Hadoop cluster to let hadoop daemon load tez class. (I deploy a
pseudo hadoop 3.3.6 cluster in my local env.)
* 3: Restart HS2 to load tez0.10.5 class
* 4: And then try to login in beeline client, but failed with exception.
> Issue related to Tez-0.10.5 rc0
> ---
>
> Key: TEZ-4630
> URL: https://issues.apache.org/jira/browse/TEZ-4630
> Project: Apache Tez
> Issue Type: Test
>Reporter: Butao Zhang
>Priority: Major
>
> Test Env: *Hadoop3.3.6* & Hive latest master branch, and Tez 0.10.5
> [https://dist.apache.org/repos/dist/dev/tez/apache-tez-0.10.5-rc0/apache-tez-0.10.5-bin.tar.gz]
>
> *NOTE:* if using {color:#de350b}*Hadoop3.4.1, the issue will disappear.*
> {color}
>
> Found that Beeline session will fail with:
> {color:#FF}java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29{color}
> {code:java}
> zbt@zbt-ubuntu:~$ /usr/tools/apache-hive-beeline-4.1.0-SNAPSHOT/bin/beeline
> -u "jdbc:hive2://127.0.0.1:1/default hive"HADOOP_HOME not set, executing
> beeline using JAVAConnecting to jdbc:hive2://127.0.0.1:1/defaultError:
> Could not open client transport with JDBC Uri:
> jdbc:hive2://100.71.9.13:10004/default: Failed to open new session:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.sun.proxy.$Proxy29 (state=08S01,code=0) {code}
> {color:#FF}Detailed stack trace in HS2 log:{color}
> {code:java}
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Client protocol version:
> HIVE_CLI_SERVICE_PROTOCOL_V10
> 2025-05-15T09:59:09,992 INFO [HiveServer2-Handler-Pool: Thread-74]
> thrift.ThriftCLIService: Creating Hive session handle for user [null] from IP
> 100.72.1.9
> 2025-05-15T09:59:09,995 WARN [HiveServer2-Handler-Pool: Thread-74]
> service.CompositeService: Failed to open session
> java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could not
> initialize class com.sun.proxy.$Proxy29
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_221]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_221]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> ~[hadoop-common-3.3.6.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at com.sun.proxy.$Proxy40.open(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.session.SessionManager.createSession(SessionManager.java:485)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:425)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:198)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:562)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:406)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1595)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1575)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.proc
