MyeoungDev commented on PR #11222: URL: https://github.com/apache/seatunnel/pull/11222#issuecomment-4857227975
Hi @davidzollo, @nzw921rx, @DanielLeens, while working on this PR, I found that the Spark E2E path may need additional Spark Translation layer support for vector types. Spark E2E executes this scenario through the Spark translation layer, and I confirmed that the current Spark translation layer does not support vector types yet. The relevant classes I found are - `org.apache.seatunnel.translation.spark.utils.TypeConverterUtils` - `org.apache.seatunnel.translation.spark.serialization.InternalRowConverter` - `org.apache.seatunnel.translation.spark.serialization.SeaTunnelRowConverter` Could you please confirm whether this understanding is correct? If so, should I add Spark Translation Layer support for vector types in this PR, or keep this PR focused on CopyTransform and open a separate PR for Spark vector type support? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
