juliomaroto commented on PR #14797: URL: https://github.com/apache/iceberg/pull/14797#issuecomment-4611665890
Hey @jerryzhujing , I've been dealing with the same issue recenty: when it comes to Snowflake compatibility with Equality Deletes into Iceberg External Tables integration on delta writes. I'd say the Snowflake incompatibility with Iceberg Equality Deletes is rather a problem of Snowflake, than a Community Issue (if you have a look to other equivalent computing engines, most of them support equality deletes, e.g. Trino, Athena, or StarRocksDB), as Snowflake states on their own public docs that they only support positional deletes. However, It would be nice/helpful if the connector could optionally include switching from equality deletes to positional deletes, which are supported by Snowflake. As per my recent investigations, not even Flink supports equality deletes on Iceberg delta writes, as performance-wise it would be subpar. Anyway, I agree that optionally including it in the connector would be a great advantage for the community, and extending the adoption of Iceberg, as Snowflake has a wide presence in the data domain in the corporate, and its b2b users base is massive. Unfortunately I don't think Snowflake will adopt soon iceberg equality deletes, maybe because for now they don't treat Iceberg as a first citizen integration, which forces to us the snowflake users to use custom CDC solutions that implement positional deletes, instead of community-wide ones, like this Sink Connector for Kafka, or the Flink one. That is said, not sure if implementing positional deletes in this Sink Connector, even optionally, is going to be accepted by the maintainers, as it's undrecommended for streaming use cases requiring deletions, such as CDC due to performance issues, as far as I understood when I reasearched about this topic in deep. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
