Hey Iceberg Nation, Here are the meeting minutes from today's meeting.
Transcription/Recording: https://youtu.be/9TRhgRq5bFk Meeting Notes: https://docs.google.com/document/d/1YuGhUdukLP5gGiqCbk0A5_Wifqe2CZWgOd3TbhY3UQg/edit#heading=h.686a3a6fup8m New Issue Template for Spec and Design Proposals: Dan created a new issue template and documentation to streamline design docs and spec updates, improving visibility into discussions and future planning in the Iceberg community. Manifest Encryption Support: Manifest encryption support has been added, further advancing Iceberg's encryption capabilities, including the ability to handle encrypted manifests. Function Pushdown in Spark: Anton enhanced function pushdown for row-level commands in Spark, refining the handling of system function pushdown for commands like merge and update. Output Spec Selection in Rewrite Data Files: Himadri introduced a feature allowing the selection of different output specs in rewrite data files, enhancing the capability for data management and optimization. Releases: 1.5.0: This version has been successfully released after numerous release candidates. 1.5.1: Potential upcoming release with discussions around schema behaviors with time travel reads, Trino integration improvements, and an extra HEAD request issue in S3 file IO. 1.6.0: Planning to include an Arrow upgrade for CVE fixes. PyIceberg 0.6.1: Focus on compatibility and bug fixes, such as addressing the creation of version one tables with non-empty partition specs and sort orders. Discussion: Spec and Design Proposals: The implementation of a new issue template and tagging system to track and manage proposals and discussions. REST Protocol Evolution: Discussion about adding capabilities to the REST protocol, ensuring client and server compatibility for new features. Schema Behavior with Time Travel Reads: Addressing the issue with schema behavior changes during time travel reads, aiming for consistency across branches. Flink Table Maintenance: A deep dive into Flink table maintenance including strategies for compaction, orphan file removal, and snapshot expiration. Discussion on handling streaming jobs, rolling manifest writers, and the RewriteFileGroupPlanner. Catalog Discussion: Deliberation on deprecating the DynamoDB catalog. How to address challenges related to the Spark 3.3 and 3.5 merge into Iceberg, specifically around dynamic partition pruning (DPP) issues.
