schenksj commented on PR #4669: URL: https://github.com/apache/datafusion-comet/pull/4669#issuecomment-4753178166
@adityavaish thanks for this PR! This is the same general approach I started with as well, just implementing the scan without the integrating the delta parquet reader. It seemed so simple at first until I discovered all the interesting things around virtual columns, nuances around column naming modes, microsecond precision dates, streaming, CDC, etc... that come up when integrating the full text case suite. Also, the team asked for the work to be fully build gated in case there is a skew between iceberg/parquet/delta that makes a version not feasible and/or there are challenges maintaining delta (its far more complex than iceberg in implementation). Since we have a complete implementation that passes all delta unit tests with very few comet declines, and we're working to merge those changes, it might make sense for your to contribute to the post-merge version if you have passion here? Would love to have the help when the time comes? (Of course I'm not a maintainer so I don't have decision making authority) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
