lgbo-ustc commented on PR #8713:
URL:
https://github.com/apache/incubator-gluten/pull/8713#issuecomment-2655270926
There is an exception
```
[2025-02-12T04:35:07.911Z] - SPARK-36536: use casting when datetime pattern
is not set *** FAILED ***
[2025-02-12T04:35:07.911Z] org.apache.spark.SparkException: Job aborted.
[2025-02-12T04:35:07.911Z] at
org.apache.spark.sql.errors.QueryExecutionErrors$.jobAbortedError(QueryExecutionErrors.scala:651)
[2025-02-12T04:35:07.911Z] at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:371)
[2025-02-12T04:35:07.911Z] at
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:195)
[2025-02-12T04:35:07.911Z] at
org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113)
[2025-02-12T04:35:07.911Z] at
org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111)
[2025-02-12T04:35:07.911Z] at
org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125)
[2025-02-12T04:35:07.911Z] at
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
[2025-02-12T04:35:07.911Z] at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
[2025-02-12T04:35:07.911Z] at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
[2025-02-12T04:35:07.911Z] at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
```
It may be releated to following time zone setting
```scala
// Add configs
import org.apache.gluten.backendsapi.clickhouse.CHConfig._
conf.setCHConfig(
"timezone" -> conf.get("spark.sql.session.timeZone",
TimeZone.getDefault.getID),
"local_engine.settings.log_processors_profiles" -> "true")
conf.setCHSettings("spark_version", SPARK_VERSION)
// add memory limit for external sort
if (conf.getLong(RuntimeSettings.MAX_BYTES_BEFORE_EXTERNAL_SORT.key, -1)
< 0) {
if (conf.getBoolean("spark.memory.offHeap.enabled", defaultValue =
false)) {
val memSize =
JavaUtils.byteStringAsBytes(conf.get("spark.memory.offHeap.size"))
if (memSize > 0L) {
val cores = conf.getInt("spark.executor.cores", 1).toLong
val sortMemLimit = ((memSize / cores) * 0.8).toLong
logDebug(s"max memory for sorting: $sortMemLimit")
conf.set(RuntimeSettings.MAX_BYTES_BEFORE_EXTERNAL_SORT.key,
sortMemLimit.toString)
}
}
}
// Load supported hive/python/scala udfs
UDFMappings.loadFromSparkConf(conf)
LOG_ERROR(getLogger("CHListenerApi"), "xxxx initNative")
CHNativeExpressionEvaluator.initNative(conf.getAll.toMap)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
