richardc-db commented on PR #50849: URL: https://github.com/apache/spark/pull/50849#issuecomment-2902973199
hmm, @cloud-fan, seems like this causes a test to fail with `org.apache.spark.sql.CacheTableInKryoSuite`. The error is ``` java.io.IOException: com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.sql.catalyst.expressions.UnsafeRow[] ``` I'm guessing this is because the inputRDD has unsafe rows? The test passes after undoing the change described [here](https://github.com/apache/spark/pull/50849#discussion_r2097521541)... do you think we should undo this change and go back to adding an unsafe projection in `doExecute`, like so: ``` protected override def doExecute(): RDD[InternalRow] = { val numOutputRows = longMetric("numOutputRows") rdd.mapPartitionsWithIndexInternal { (_, iter) => val proj = UnsafeProjection.create(schema) iter.map { r => numOutputRows += 1 proj(r) } } } ``` and setting `createUnsafeProjection: Boolean = true` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
