viirya commented on PR #53276: URL: https://github.com/apache/spark/pull/53276#issuecomment-3624317414
> One more question I forgot. Do we need to make `VariantExtraction` extend `Serializable`? Is it supposed to be sent to executors? Seems like this will be done on the driver. Hmm, for the built-in ParquetScan, it is not required because it doesn't send `VariantExtraction` to executors. It uses the extraction info to transform schema and the physical reader uses the schema to do variant rewriting. But probably it is required to be `Serializable` for third-party datasource implementions as we don't know how they will use the extraction info. They may send the info to executors and do rewriting there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
