[
https://issues.apache.org/jira/browse/SPARK-38315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17497362#comment-17497362
]
Max Gekk commented on SPARK-38315:
----------------------------------
I am working on this.
> Add a config to collect objects as Java 8 types in the Thrift server
> --------------------------------------------------------------------
>
> Key: SPARK-38315
> URL: https://issues.apache.org/jira/browse/SPARK-38315
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Major
>
> Add new config that should control collect(), and allow to enable/disable to
> Java 8 types in the Thrift server. The config should solve the following
> issue:
> When an user connects to the Thrift Server and a query involve a datasource
> connect which doesn't handle Java8 types, the user observes the following
> exception:
> {code:java}
> ERROR SparkExecuteStatementOperation: Error executing query with
> ac61b10a-486e-463b-8726-3b61da58582e, currentState RUNNING,
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
> stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0
> (TID 8) (10.157.1.194 executor 0): java.lang.RuntimeException: Error while
> encoding: java.lang.RuntimeException: java.sql.Timestamp is not a valid
> external type for schema of timestamp
> if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null
> else staticinvoke(class org.apache.spark.sql.catalyst.util.DateTimeUtils$,
> TimestampType, instantToMicros,
> validateexternaltype(getexternalrowfield(assertnotnull(input[0,
> org.apache.spark.sql.Row, true]), 0, loan_perf_date), TimestampType), true,
> false) AS loan_perf_date#1125
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:239)
>
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:210)
>
> at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
