comphead commented on code in PR #5368:
URL: https://github.com/apache/datafusion-comet/pull/5368#discussion_r3883477086
##########
spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala:
##########
@@ -112,10 +112,10 @@ case class EliminateRedundantTransitions(session:
SparkSession)
// 4.1+ matches the renamed `MapInArrowExec`.
//
// Falls back to vanilla Spark when
`spark.sql.execution.arrow.useLargeVarTypes` is enabled:
- // CometArrowPythonRunnerBase.copyVector does raw `setBytes` on each
Arrow buffer, but Comet's
- // source string/binary vectors always use 4-byte offsets while the
destination root is
- // allocated with 8-byte offsets when this conf is on. The buffer counts
match but the
- // offset width does not, so a direct memcpy would corrupt the offsets.
+ // Native Comet string/binary vectors use 4-byte offsets. The IPC schema
follows these
Review Comment:
not sure why this was changed, it doesn't relate to PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
