HeartSaVioR edited a comment on pull request #31771: URL: https://github.com/apache/spark/pull/31771#issuecomment-794498033
I guess we haven't decided the important things. Let's stop making progress and decide before continue. 1. Would we want to pull the dependency from Confluent Schema Registry? There's no issue on license, but I also see we are trying to be "vendor-neutral" via avoiding vendor related things directly. No preference on this as I might not know about some decision being taken before. 2. If the answer for 1 is yes (meaningless if the answer is no), given SR is only working with Kafka topic, are we sure doing it via `from_avro`/`to_avro` is the right way? I'd rather say Spark SQL Kafka should be smart to support schema based on SR instead, but I agree it requires a bunch of hurdles, so it's OK if we just like to go through `from_avro` / `to_avro`. 3. If the answer for 1 is yes (same, meaningless if the answer is no), where we'd like to add the dependency? We've been concerned about adding external artifacts into core/sql so my assumption is to add to external. That said, `from_avro`/`to_avro` wouldn't be the place to extend. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
