snazy commented on PR #1190: URL: https://github.com/apache/polaris/pull/1190#issuecomment-2745199009
@gh-yzou, I'm not against the Spark plugin, my main concern is how we manage the complexity. I'm supportive on the idea of putting this tool in a different repo. This is also what a lot of other OSS projects do for implementations and tools. The Spark plugin is independent from the "main" Polaris code base. So putting it into a different repo and concentrate on the _very specific_ needs for Spark with its Scala dependencies makes more sense to me. It also offers the opportunity to release the plugin at a different pace. From experience with the Nessie Spark plugin I can say that it's better to have the Spark plugin in a different build/repo. Integration tests become extremely difficult, especially if both Polaris and the Spark plugin share the same (Maven) group ID. Further, there may also be a need to support multiple Iceberg versions, so the matrix explodes exponentially: * Polaris versions * Iceberg versions * Spark versions * Scala versions To me, that ^ isn't manageable in the main Polaris repo. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@polaris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
