Max Gekk created SPARK-38315:
--------------------------------
Summary: Add a config to collect objects as Java 8 types in the
Thrift server
Key: SPARK-38315
URL: https://issues.apache.org/jira/browse/SPARK-38315
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.3.0
Reporter: Max Gekk
Assignee: Max Gekk
Add new config that should control collect(), and allow to enable/disable to
Java 8 types in the Thrift server. The config should solve the following issue:
When an user connects to the Thrift Server and a query involve a datasource
connect which doesn't handle Java8 types, the user observes the following
exception:
{code:java}
ERROR SparkExecuteStatementOperation: Error executing query with
ac61b10a-486e-463b-8726-3b61da58582e, currentState RUNNING,
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID
8) (10.157.1.194 executor 0): java.lang.RuntimeException: Error while encoding:
java.lang.RuntimeException: java.sql.Timestamp is not a valid external type for
schema of timestamp
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else
staticinvoke(class org.apache.spark.sql.catalyst.util.DateTimeUtils$,
TimestampType, instantToMicros,
validateexternaltype(getexternalrowfield(assertnotnull(input[0,
org.apache.spark.sql.Row, true]), 0, loan_perf_date), TimestampType), true,
false) AS loan_perf_date#1125
at
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:239)
at
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:210)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
