Me too. It's 1 AM–2 AM GMT+8 for me. The best I can do is 0AM-1AM 😪 like the other syncs. Is it possible to move the sync meeting one or two hours earlier?
On Wed, Oct 21, 2026 at 2:21 PM Eduard Tudenhöfner <[email protected]> wrote: > Hey Anurag, > > I would be generally interested in participating in those Spark syncs, but > the time slot is too late in the evening (7PM-8PM CET) for me personally > and doesn't work for me. > I also don't have a better suggestion for increasing involvement, given > that everyone is distributed across different time zones. > > Eduard > > On Wed, Oct 21, 2026 at 1:17 AM Anurag Mantripragada < > [email protected]> wrote: > >> Hi everyone, >> >> We've noticed low turnout recently for the Iceberg Spark community syncs >> <https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?tab=t.0#heading=h.r977qio1wsv2>, >> especially from Spark stakeholders, even with exciting ongoing work on new >> features like Spark 4.2. >> >> I still find these syncs super valuable (which was the main reason for >> starting them) because they provide: >> >> - Dedicated time for Spark-specific topics that don't fit into the >> main community sync >> - Space to talk through Spark-specific roadmaps and related work >> - Better coordination for Spark stakeholders >> >> >> I would like to understand if the current time isn't convinent or if >> these syncs aren't that valuable. Please provide any feedback that will >> help us decide whether to reschedule or cancel these altogether. >> >> ~ Anurag >> >> >> On Tue, Feb 17, 2026 at 10:42 AM Anurag Mantripragada < >> [email protected]> wrote: >> >>> Hi all, >>> >>> Thanks for joining the Iceberg-Spark community sync today. You can find >>> the recording here <https://youtu.be/vNvOHMHXHGw>. >>> >>> Below are the highlights from the meeting: >>> >>> - Comet Integration: Thanks to everyone involved in reviewing and >>> merging the FileFormat API PRs to unblock the Comet team. The final >>> required PR is #15328 <https://github.com/apache/iceberg/pull/15328>. >>> - Variant Support: There are questions regarding the status of the >>> variant support PR (#14297 >>> <https://github.com/apache/iceberg/pull/14297>), specifically >>> whether it completes the full end-to-end implementation for reading and >>> writing variants in Spark and if it will be included in the next release. >>> - DSV2 Versions Framework: We discussed the DSV2 version framework >>> PRs and whether they are required for the upcoming release. >>> >>> ~ Anurag >>> >>> On Tue, Jan 20, 2026 at 1:49 PM Aihua Xu <[email protected]> wrote: >>> >>>> Thanks Anurag for driving this and included Support writing shredded >>>> variant in Iceberg-Spark (#14297 >>>> <https://github.com/apache/iceberg/pull/14297>). Appreciate the folks >>>> from Spark side can help review the PR. >>>> >>>> Thanks, >>>> Aihua >>>> >>>> >>>> >>>> On Tue, Jan 20, 2026 at 1:17 PM karuppayya <[email protected]> >>>> wrote: >>>> >>>>> Thanks Anurag for driving this. Unfortunately, I wasnt able to attend >>>>> today. I reviewed the video. >>>>> >>>>> @prashant, regarding >>>>> >>>>> - >>>>> >>>>> Alpha family aggregate support - #52551 >>>>> <https://github.com/apache/spark/pull/52551> >>>>> >>>>> The Iceberg spec[1] requires that NDV computed via Alpha sketch. We >>>>> currently have a ThetaAgg expression in the Iceberg code that uses the >>>>> Alpha family[2]. >>>>> Spark recently introduced support for ThetaSketch aggregates[3] (*which >>>>> currently only supports the quickselect family*), this PR introduces >>>>> *Alpha >>>>> family support*. >>>>> Once we have this, we don't need to maintain the code in Iceberg and >>>>> also get the benefits from Spark community(either with improvement to >>>>> Spark >>>>> Catalyst in general or the expression specifically) >>>>> >>>>> With regards to >>>>> >>>>> - >>>>> >>>>> Codegen for MergeRowsExec - #52399 >>>>> <https://github.com/apache/spark/pull/52399> >>>>> >>>>> This would help speed up merge execution (and also simplify >>>>> SplitIterator >>>>> <https://github.com/apache/spark/pull/52399/changes#diff-a572ff40254b26b4a903f101ee466dd2dff9b8c7954a3b957fe5fc25b87ee10aR241-R242> >>>>> logic). >>>>> >>>>> - Karuppayya >>>>> [1] - >>>>> https://iceberg.apache.org/puffin-spec/#apache-datasketches-theta-v1-blob-type >>>>> [2] - >>>>> https://github.com/apache/iceberg/blob/main/spark/v4.1/spark/src/main/scala/org/apache/spark/sql/stats/ThetaSketchAgg.scala#L66-L68 >>>>> [3] - >>>>> https://github.com/karuppayya/spark/commit/6ff9edcaf16c90007508f15de98fac361e234381 >>>>> >>>>> >>>>> >>>>> >>>>> On Tue, Jan 20, 2026 at 12:28 PM Anurag Mantripragada < >>>>> [email protected]> wrote: >>>>> >>>>>> Thanks everyone for joining the first Iceberg/Spark community sync. >>>>>> >>>>>> Here is the recording: >>>>>> https://youtu.be/g4n2hwdFosE?si=n9hVRhCThshuOqd5 >>>>>> >>>>>> >>>>>> Below are the discussion highlights. >>>>>> >>>>>> Datafusion Comet integration >>>>>> >>>>>> - >>>>>> - >>>>>> >>>>>> Spark: Encapsulate parquet objects for Comet (#13786 >>>>>> <https://github.com/apache/iceberg/pull/13786>) >>>>>> - >>>>>> >>>>>> Future of Iceberg support in Comet (datafusion-comet#2921 >>>>>> <https://github.com/apache/datafusion-comet/issues/2921>) >>>>>> - >>>>>> >>>>>> Mailing List Discussion >>>>>> >>>>>> <https://lists.apache.org/thread/vr9nsbd5nhg3d20nmtyj4b3zsw9229gd> >>>>>> - >>>>>> >>>>>> Notes: >>>>>> - >>>>>> >>>>>> Rust vs Java - Discuss and vote in the dev list >>>>>> - >>>>>> >>>>>> To move forward with (#13786 >>>>>> <https://github.com/apache/iceberg/pull/13786>) - >>>>>> Discuss in FileFormat API sync if there are any pending >>>>>> items this PR needs >>>>>> updates on. >>>>>> - >>>>>> >>>>>> Make a decision to merge the PR vs waiting for >>>>>> FileFormat API >>>>>> >>>>>> >>>>>> - >>>>>> >>>>>> Spark 3.4 Deprecation >>>>>> - >>>>>> >>>>>> Spark: Remove Spark 3.4 support (#14122 >>>>>> <https://github.com/apache/iceberg/pull/14122>) >>>>>> - >>>>>> >>>>>> Notes: >>>>>> - >>>>>> >>>>>> Wait until comet integration is resolved. >>>>>> >>>>>> - >>>>>> >>>>>> Spark 4.1/4.2 >>>>>> - >>>>>> >>>>>> Spark: Add support for 4.2.0-preview (#14984 >>>>>> <https://github.com/apache/iceberg/pull/14984>) >>>>>> - >>>>>> >>>>>> Spark 4.1: Initial support for MERGE INTO schema evolution >>>>>> (#14970 <https://github.com/apache/iceberg/pull/14970>) >>>>>> - >>>>>> >>>>>> Notes: >>>>>> - >>>>>> >>>>>> 4.1 is the current latest version. New PRs must go to it >>>>>> - >>>>>> >>>>>> Spark 4.1 introduces a version framework. Anton is >>>>>> working on integrating it with Iceberg. This greatly >>>>>> simplifies Iceberg >>>>>> lifecycle management but requires non-trivial integration >>>>>> work. >>>>>> - >>>>>> >>>>>> Prefer not to make any releases with 4.1 until this is >>>>>> in. >>>>>> >>>>>> >>>>>> - >>>>>> >>>>>> DSv2 and sort order reporting >>>>>> - >>>>>> >>>>>> Spark (4.0, 3.5): Set data file sort_order_id in manifest >>>>>> for writes from Spark (#14683 >>>>>> <https://github.com/apache/iceberg/pull/14683>) >>>>>> - >>>>>> >>>>>> The rebase has many changes. Ask author to fix. >>>>>> - >>>>>> >>>>>> Spark 4.0: Implement SupportsReportOrdering DSv2 API (#14948 >>>>>> <https://github.com/apache/iceberg/pull/14948>) >>>>>> - >>>>>> >>>>>> Move to 4.1 for easier review >>>>>> >>>>>> >>>>>> - >>>>>> >>>>>> Compaction/Table maintenance/DR >>>>>> - >>>>>> >>>>>> Spark 4.0: RewriteTablePath support for multiple source >>>>>> and destination prefixes (#14355 >>>>>> <https://github.com/apache/iceberg/pull/14355>) >>>>>> - >>>>>> >>>>>> Spark 4.0: Optional switch to log expire data files during >>>>>> ExpireSnapshots action (#14354 >>>>>> <https://github.com/apache/iceberg/pull/14354>) >>>>>> - >>>>>> >>>>>> Notes: >>>>>> - >>>>>> >>>>>> Trace level logging >>>>>> - >>>>>> >>>>>> How about logging it to another Iceberg table? >>>>>> - >>>>>> >>>>>> Use the dataframe of files and log separately? >>>>>> >>>>>> >>>>>> - >>>>>> >>>>>> V3 spec implementation >>>>>> - >>>>>> >>>>>> Spark: Support writing shredded variant in Iceberg-Spark ( >>>>>> #14297 <https://github.com/apache/iceberg/pull/14297>) >>>>>> - >>>>>> >>>>>> Notes: >>>>>> - >>>>>> >>>>>> Status of Variant type support - consolidate and track >>>>>> somewhere >>>>>> - >>>>>> >>>>>> Filter pushdown not implemented >>>>>> - >>>>>> >>>>>> The write support PR is new, will review. It should have >>>>>> Iceberg metadata changes to indicate the variant shredding >>>>>> so Spark can use >>>>>> it. >>>>>> - >>>>>> >>>>>> #14297 <https://github.com/apache/iceberg/pull/14297> >>>>>> Will be reviewed >>>>>> >>>>>> >>>>>> - >>>>>> >>>>>> Spark UDF Support >>>>>> - >>>>>> >>>>>> SQL UDF support Stage 1 (#14954 >>>>>> <https://github.com/apache/iceberg/pull/14954>) (The >>>>>> corresponding Spark SPIP: SPIP: Catalog-backed Code-Literal >>>>>> Functions (SQL and Python) with Catalog SPI and CRUD >>>>>> >>>>>> <https://docs.google.com/document/d/186cTAZxoXp1p8vaSunIaJmVLXcPR-FxSiLiDUl8kK8A/edit?tab=t.0#heading=h.for1fb3tezo3> >>>>>> ) >>>>>> - >>>>>> >>>>>> Notes: >>>>>> - >>>>>> >>>>>> Waiting for the proposal vote and spark side SPIP >>>>>> related to this. >>>>>> - >>>>>> >>>>>> Spark 4.0: Spark UDF POC (#14505 >>>>>> <https://github.com/apache/iceberg/pull/14505>) >>>>>> - >>>>>> >>>>>> Huaxin to delete this PR. This version is hacky. >>>>>> >>>>>> >>>>>> - >>>>>> >>>>>> DDL and schema evolution >>>>>> - >>>>>> >>>>>> CREATE TABLE LIKE support (#14269 >>>>>> <https://github.com/apache/iceberg/pull/14269>) >>>>>> - >>>>>> >>>>>> Notes: >>>>>> - >>>>>> >>>>>> Recommend not to add SQL extensions in Iceberg code >>>>>> anymore. >>>>>> - >>>>>> >>>>>> They are fragile and need maintenance and have to >>>>>> work well with Spark >>>>>> - >>>>>> >>>>>> Alternatively, consider writing a procedure to do this >>>>>> until Spark has native support. >>>>>> - >>>>>> >>>>>> Native Spark support for CREATE TABLE LIKE is not yet >>>>>> implemented. >>>>>> Spark PRs >>>>>> - >>>>>> >>>>>> Alpha family aggregate support - #52551 >>>>>> <https://github.com/apache/spark/pull/52551> >>>>>> - >>>>>> >>>>>> Notes: >>>>>> - >>>>>> >>>>>> Okay to have Spark only changes that can potentially >>>>>> help in Iceberg use-cases >>>>>> - >>>>>> >>>>>> Elaborate on the use of this? How does this integrate >>>>>> with Iceberg? >>>>>> >>>>>> >>>>>> - >>>>>> >>>>>> Codegen for MergeRowsExec - #52399 >>>>>> <https://github.com/apache/spark/pull/52399> >>>>>> - >>>>>> >>>>>> Notes: >>>>>> - >>>>>> >>>>>> This is a heavily used Exec node in Iceberg so this is >>>>>> good to have. >>>>>> - >>>>>> >>>>>> The community will review this >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> ~Anurag >>>>>> >>>>>> On Thu, Jan 15, 2026 at 6:48 PM Anton Okolnychyi < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> If anyone has long-standing PRs related to Spark, it may be a good >>>>>>> forum to get some reviews and help from the community. >>>>>>> >>>>>>> ср, 14 січ. 2026 р. о 11:23 Anurag Mantripragada < >>>>>>> [email protected]> пише: >>>>>>> >>>>>>>> Thanks Kevin, >>>>>>>> >>>>>>>> All, please review the doc >>>>>>>> <https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?tab=t.0> >>>>>>>> and >>>>>>>> add any agenda items I may have missed. See you on Tuesday. >>>>>>>> >>>>>>>> ~ Anurag >>>>>>>> >>>>>>>> On Wed, Jan 14, 2026 at 11:20 AM Kevin Liu <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Connected with Anurag on Slack. I just added a new event to the >>>>>>>>> Iceberg Dev calendar for next week Tuesday Jan 20th from 10AM - 11AM >>>>>>>>> PT, "*Iceberg >>>>>>>>> - Spark Community Sync*". It's a monthly recurring meeting and >>>>>>>>> the google meets link is set to open to the public. >>>>>>>>> Happy to make changes based on feedback. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Kevin Liu >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jan 14, 2026 at 10:57 AM Kevin Liu <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Looking at the current Iceberg dev calendar schedule, we have a >>>>>>>>>> slot next week Tuesday or Friday for a monthly recurring sync. >>>>>>>>>> Wednesday >>>>>>>>>> corresponds with the main Community Sync in some weeks. >>>>>>>>>> Please let me know the preferred day and time and I can help set >>>>>>>>>> it up! >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Kevin Liu >>>>>>>>>> >>>>>>>>>> On Tue, Jan 13, 2026 at 10:58 AM Anurag Mantripragada < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Kevin, >>>>>>>>>>> >>>>>>>>>>> I'm open to ideas, but I think we could start with monthly >>>>>>>>>>> cadence for Spark syncs and increase the frequency if the community >>>>>>>>>>> feels >>>>>>>>>>> we need to meet more often. Could you please set up a time on the >>>>>>>>>>> Iceberg >>>>>>>>>>> dev calendar? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Anurag >>>>>>>>>>> >>>>>>>>>>> On Fri, Jan 9, 2026 at 10:16 AM Anurag Mantripragada < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thanks Anton and Kevin, >>>>>>>>>>>> >>>>>>>>>>>> I wrote a doc with general themes from the Spark PRs and Issues >>>>>>>>>>>> I browsed in the repo. Please feel free to add more if I may have >>>>>>>>>>>> missed >>>>>>>>>>>> anything. >>>>>>>>>>>> >>>>>>>>>>>> https://docs.google.com/document/d/19nno1RoPznbbxKOZZddZNHHafa7XULjbN6RPExdr2n4/edit?tab=t.0 >>>>>>>>>>>> >>>>>>>>>>>> Looking forward to meeting you all and talking about all things >>>>>>>>>>>> Spark! >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Anurag >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Jan 9, 2026 at 10:03 AM Kevin Liu < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> +1 great idea! >>>>>>>>>>>>> Let's start a doc with potential discussion items and find a >>>>>>>>>>>>> time on the calendar. I have permission to add events to the >>>>>>>>>>>>> "iceberg dev >>>>>>>>>>>>> events" calendar. Happy to help with the logistics once the time >>>>>>>>>>>>> and >>>>>>>>>>>>> cadence is decided. >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> Kevin Liu >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Jan 7, 2026 at 4:35 PM Anton Okolnychyi < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> YES! I have been meaning to suggest the same. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can you start a doc with the pool of items to which everyone >>>>>>>>>>>>>> can contribute to? >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Anton >>>>>>>>>>>>>> >>>>>>>>>>>>>> ср, 7 січ. 2026 р. о 15:30 Anurag Mantripragada < >>>>>>>>>>>>>> [email protected]> пише: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi folks, happy new year! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (Sorry if I sent this email more than once, my attempts of >>>>>>>>>>>>>>> sending this from a different email failed) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> There are a few Spark changes the community is working on >>>>>>>>>>>>>>> including >>>>>>>>>>>>>>> - Sort order reporting [1], [2] >>>>>>>>>>>>>>> - Spark 4.1 support [3] >>>>>>>>>>>>>>> - Future of Datafusion-Comet support [4] [5] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Community members interested in the Spark integration have >>>>>>>>>>>>>>> been discussing it in smaller groups. However, we believe that >>>>>>>>>>>>>>> the general >>>>>>>>>>>>>>> community sync should include all updates, and discussing >>>>>>>>>>>>>>> Spark-specific >>>>>>>>>>>>>>> matters may not be the most effective use of that sync. I was >>>>>>>>>>>>>>> wondering if >>>>>>>>>>>>>>> it will be useful to create a Spark-Iceberg >>>>>>>>>>>>>>> integration-specific sync on >>>>>>>>>>>>>>> the calendar, similar to what we have for individual proposals. >>>>>>>>>>>>>>> This sync >>>>>>>>>>>>>>> will not replace the community sync, which will still be used >>>>>>>>>>>>>>> for broader >>>>>>>>>>>>>>> discussions including any new spark topics that come out of the >>>>>>>>>>>>>>> spark sync. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If there’s interest in doing these spark breakout syncs, I’m >>>>>>>>>>>>>>> happy to volunteer to run them. Please let me know what you all >>>>>>>>>>>>>>> think. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> ~ Anurag >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] - https://github.com/apache/iceberg/pull/14683 >>>>>>>>>>>>>>> [2] - https://github.com/apache/iceberg/pull/14948 >>>>>>>>>>>>>>> [3] - https://github.com/apache/iceberg/pull/14970 >>>>>>>>>>>>>>> [4] - https://github.com/apache/datafusion-comet/issues/2921 >>>>>>>>>>>>>>> [5] - >>>>>>>>>>>>>>> https://lists.apache.org/thread/vr9nsbd5nhg3d20nmtyj4b3zsw9229gd >>>>>>>>>>>>>>> >>>>>>>>>>>>>>
