Shex created SPARK-32070:
----------------------------
Summary: Spark 3.0 ERROR TThreadPoolServer
Key: SPARK-32070
URL: https://issues.apache.org/jira/browse/SPARK-32070
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.0.0
Reporter: Shex
I'm using the latest spark 3.0.0 with simple settings, after loading files
with beeline (bin/beeline -u jdbc:hive2://127.0.0.1:10000/ -f load.view), the
message has appeared in Thrift's log.
Also, when query is executed (select * from load; ) at the end of execution in
the Thrift's log there is also the same message.
Even when you connect with beeline (bin/beeline -u
jdbc:hive2://127.0.0.1:10000/) and do "!quit" the same message is generated.
Queries with beeline are working fine, just there is the same message.
Error message:
ERROR TThreadPoolServer: Thrift error occurred during processing of message.
org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374)
at
org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433)
at
org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:43)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:53)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I've tried to run Spark master and slave version 3.0.0 and Thrift from Spark
2.4.5, I got the same message.
Config is simple and used for testing before production:
spark.driver.extraClassPath=/var/opt/engine/spark/mysql-connector-java-8.0.20.jar
spark.executor.extraClassPath=/var/opt/engine/spark/mysql-connector-java-8.0.20.jar
spark.driver.memory=30g
spark.driver.cores=12
spark.executor.cores=12
spark.executor.instances=1
spark.executor.memory=30g
spark.network.timeout=1800
spark.rpc.askTimeout=6000
spark.rpc.lookupTimeout=3000
spark.sql.hive.thriftServer.singleSession=true
spark.sql.warehouse.dir=/var/opt/engine/spark
spark.hadoop.hive.metastore.warehouse.dir=/var/opt/engine/spark
hive.metastore.warehouse.dir=/var/opt/engine/spark
user.dir=/var/opt/engine/spark
derby.system.home=/var/opt/engine/spark
SPARK_WORKER_MEMORY=32g
SPARK_WORKER_CORES=12
SPARK_WORKER_INSTANCES=1
SPARK_LOCAL_IP=10.10.10.10
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
