Hi: Sorry I couldn't join the meeting due to timezone constraints.
I want to add some summary about progress of iceberg-rust: 1. We have added hive catalog support(except update table methods). Thanks Xuanwo and Marvin Lanhenke(github id: marvinlanhenke)! 2. The glue catalog is in progress. Thanks Marvin Lanhenke(github id: marvinlanhenke)! 3. Some enhancement of rest catalog, such as oauth, custom headers. Thanks whynick1, TennyZhuang! 4. The community is actively working on reading support, including reading to arrow, filter support, please stay tuned! 5. We have more and more contributors involved in the development, and the number of contributors has reached 36! 6. I have created a github project to help schedule the development so that contributors could easily pick up issues they are interested in! cc Brian, I could update the progress to meeting notes if necessary. On Thu, Mar 28, 2024 at 6:50 AM Brian Olsen <bitsondata...@gmail.com> wrote: > Hey Iceberg Nation, > > Here are the meeting minutes from today's meeting. > > Transcription/Recording: https://youtu.be/9TRhgRq5bFk > Meeting Notes: > https://docs.google.com/document/d/1YuGhUdukLP5gGiqCbk0A5_Wifqe2CZWgOd3TbhY3UQg/edit#heading=h.686a3a6fup8m > > > New Issue Template for Spec and Design Proposals: Dan created a new issue > template and documentation to streamline design docs and spec updates, > improving visibility into discussions and future planning in the Iceberg > community. > Manifest Encryption Support: Manifest encryption support has been added, > further advancing Iceberg's encryption capabilities, including the ability > to handle encrypted manifests. > Function Pushdown in Spark: Anton enhanced function pushdown for row-level > commands in Spark, refining the handling of system function pushdown for > commands like merge and update. > Output Spec Selection in Rewrite Data Files: Himadri introduced a feature > allowing the selection of different output specs in rewrite data files, > enhancing the capability for data management and optimization. > > Releases: > 1.5.0: This version has been successfully released after numerous release > candidates. > 1.5.1: Potential upcoming release with discussions around schema behaviors > with time travel reads, Trino integration improvements, and an extra HEAD > request issue in S3 file IO. > 1.6.0: Planning to include an Arrow upgrade for CVE fixes. > PyIceberg 0.6.1: Focus on compatibility and bug fixes, such as addressing > the creation of version one tables with non-empty partition specs and sort > orders. > > Discussion: > Spec and Design Proposals: The implementation of a new issue template and > tagging system to track and manage proposals and discussions. > REST Protocol Evolution: Discussion about adding capabilities to the REST > protocol, ensuring client and server compatibility for new features. > Schema Behavior with Time Travel Reads: Addressing the issue with schema > behavior changes during time travel reads, aiming for consistency across > branches. > Flink Table Maintenance: A deep dive into Flink table maintenance > including strategies for compaction, orphan file removal, and snapshot > expiration. Discussion on handling streaming jobs, rolling manifest > writers, and the RewriteFileGroupPlanner. > Catalog Discussion: Deliberation on deprecating the DynamoDB catalog. > How to address challenges related to the Spark 3.3 and 3.5 merge into > Iceberg, specifically around dynamic partition pruning (DPP) issues. >
