Sorry to clarify my position on the timestamp nano fix, I think as long as we're failing and not in a position to silently produce incorrect values I'm OK to not block on a fix for that.
On Mon, Jul 7, 2025 at 5:19 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > Thanks Steven, > > I agree that we can defer the timestamp nano fix, I had to write some > tests to prove it to myself but every long microseconds or string timestamp > value that's within range would always fail with an exception when > converting to nanos; so I think we're OK as is though that is a long > standing issue that'd be good to address to unblock Trino on supporting the > new precision. > > I also added the Spark 4.0 row lineage support PR > <https://github.com/apache/iceberg/pull/13310> to the milestone since > folks who may be upgrading from 3.4/3.5 to 4.0 would go from a system which > is correctly capturing lineage information to a system which will just > start dropping those. My biggest concern is that for these users, this is > effectively a regression where they would stop trusting lineage data. > > I've asked folks to take a look, and encourage everyone to leave their > feedback on the PR so I can quickly address it and we can move forward with > the release. > > Thanks, > > Amogh Jahagirdar > > On Mon, Jul 7, 2025 at 1:07 PM Steven Wu <stevenz...@gmail.com> wrote: > >> Updates for the remaining 3 open PRs for 1.10.0 milestone >> https://github.com/apache/iceberg/milestone/54 >> >> Core: Keep track of data files to be removed for orphaned DV detection >> <https://github.com/apache/iceberg/pull/13222> >> * got 2 approvals. should be merged in a day or two >> >> Spark 4.0: Migrate Iceberg Stored Procedures to Spark built-in >> implementations <https://github.com/apache/iceberg/pull/13106> >> * Got 3 approvals. expect it to be merged in a day or two >> >> Core: Fix numeric overflow of timestamp nano literal >> <https://github.com/apache/iceberg/pull/11775> >> * It breaks the use case of long literal value for nano timestamp column. >> But there is no correctness issue. Hence I am favoring moving it out of the >> 1.10.0 milestone >> * there is no consensus on the path forward yet. >> >> On Thu, Jul 3, 2025 at 2:28 PM Szehon Ho <szehon.apa...@gmail.com> wrote: >> >>> Thanks Steven! >>> >>> >>> On Jul 3, 2025, at 1:56 PM, Steven Wu <stevenz...@gmail.com> wrote: >>> >>> >>> > because it makes a bit of backward incompatible change by using the >>> Spark behavior (for argument case sensitivity and type coercion support), >>> we should do it in 1.10 release as we will have Spark 4.0 as part of it >>> iiuc. Else 1.11 will be backward incompatible for the Spark 4.0 jar. >>> >>> Szehon, thanks for bringing up PR 13106. I agree with your argument >>> here. It will be great to include this in the 1.10.0 release to avoid a >>> behavior change later because Spark 4.0 support was released in this >>> version. It seems close as you already approved it. Can we get a couple >>> more approvals? >>> >>> >>> >>> On Thu, Jul 3, 2025 at 1:37 PM Szehon Ho <szehon.apa...@gmail.com> >>> wrote: >>> >>>> Hi Steven >>>> >>>> Thanks. One more, what do we think about having >>>> https://github.com/apache/iceberg/pull/13106/ as part of 1.10 >>>> release? It's migrating Spark procedure to use Spark 4's new DSV2 >>>> Procedure framework. >>>> >>>> Its not a blocker but I feel, because it makes a bit of backward >>>> incompatible change by using the Spark behavior (for argument case >>>> sensitivity and type coercion support), we should do it in 1.10 release as >>>> we will have Spark 4.0 as part of it iiuc. Else 1.11 will be backward >>>> incompatible for the Spark 4.0 jar. >>>> >>>> Thanks, >>>> Szehon >>>> >>>> >>>> >>>> On Thu, Jul 3, 2025 at 1:17 PM Steven Wu <stevenz...@gmail.com> wrote: >>>> >>>>> Szehon's backport PR has been merged. Another blocker (dangling DVs >>>>> for rewrite) was also merged. >>>>> Core, Spark: Propagate orphaned delete files when rewriting data files >>>>> <https://github.com/apache/iceberg/pull/13245> >>>>> >>>>> There are still 3 blockers left, as a new PR (also danglinge DV >>>>> related) is identified as a blocker. >>>>> >>>>> Core: Keep track of data files to be removed for orphaned DV detection >>>>> <https://github.com/apache/iceberg/pull/13222> >>>>> * being actively reviewed and worked on. >>>>> >>>>> S3: Add LegacyMd5Plugin to S3 client builder >>>>> <https://github.com/apache/iceberg/pull/12264> >>>>> *being actively reviewed and worked on. >>>>> >>>>> Core: Fix numeric overflow of timestamp nano literal >>>>> <https://github.com/apache/iceberg/pull/11775> >>>>> * It causes the number overflow error for engines who have nano >>>>> precision long literal. Engines have to find workarounds currently. >>>>> Potential solutions are discussed in the PR. E.g. provide a public >>>>> method to construct a TimestampNanoLiteral so that engines/clients can use >>>>> it. >>>>> * I think it is a non-blocker for the 1.10 release. Treating the >>>>> long literal always as micro (current behavior) is not a correctness bug. >>>>> It is an important issue to be fixed so that engines can support nano >>>>> timestamp literal without going through the hoops. >>>>> >>>>> Thanks, >>>>> Steven >>>>> >>>>> On Wed, Jul 2, 2025 at 11:53 AM Szehon Ho <szehon.apa...@gmail.com> >>>>> wrote: >>>>> >>>>>> Thanks Steven for driving the release. >>>>>> >>>>>> I like to get in one more bug fix: >>>>>> https://github.com/apache/iceberg/pull/13448, it is a backport of >>>>>> https://github.com/apache/iceberg/pull/13435 (merged by Amogh) as I >>>>>> missed to do Spark 3.4, so also should be close. >>>>>> >>>>>> Thanks >>>>>> Szehon >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Jul 2, 2025 at 11:17 AM Steven Wu <stevenz...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> During today's community sync meeting, there are 2 PRs that are >>>>>>> flagged as potential blockers. I have pushed other PRs to the next >>>>>>> 1.11.0 >>>>>>> milestone. These 3 PRs are the only open items left in the 1.10.0 >>>>>>> milestone. >>>>>>> https://github.com/apache/iceberg/milestone/54 >>>>>>> >>>>>>> S3: Add LegacyMd5Plugin to S3 client builder >>>>>>> <https://github.com/apache/iceberg/pull/12264> >>>>>>> * Need community's help on reviewing this PR. Does the current >>>>>>> change solve the issue? >>>>>>> * If this still requires more significant work, we may want to >>>>>>> roll back the aws-sdk version (like we did for the 1.9.0 release) >>>>>>> >>>>>>> Core: Fix numeric overflow of timestamp nano literal >>>>>>> <https://github.com/apache/iceberg/pull/11775> >>>>>>> * The overflow error can be confusing. Is there any metadata or >>>>>>> correctness issue due to this bug? If not, we can consider it as a >>>>>>> non-blocker. >>>>>>> >>>>>>> Core, Spark: Propagate orphaned delete files when rewriting data >>>>>>> files <https://github.com/apache/iceberg/pull/13245> >>>>>>> * Very close. should be merged very soon >>>>>>> >>>>>>> It is not saying that we can't get other changes in the 1.10.0 >>>>>>> release. Just that these 3 PRs are considered potential blockers. >>>>>>> >>>>>>> Thanks, >>>>>>> Steven >>>>>>> >>>>>>> On Tue, Jul 1, 2025 at 10:38 PM Jean-Baptiste Onofré < >>>>>>> j...@nanthrax.net> wrote: >>>>>>> >>>>>>>> Hi >>>>>>>> >>>>>>>> I'm updated the PR about multi-args transforms today, but not sure I >>>>>>>> will have reviews before 1.10.0. Let's try as best effort for 1.10, >>>>>>>> else we will include in 1.11. >>>>>>>> >>>>>>>> Regards >>>>>>>> JB >>>>>>>> >>>>>>>> On Tue, Jul 1, 2025 at 6:42 PM Steven Wu <stevenz...@gmail.com> >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > Hi, >>>>>>>> > >>>>>>>> > I plan to cut a release branch in the next 1 or 2 days. >>>>>>>> > >>>>>>>> > Waiting for this row lineage related PR (and its 3.4 backport >>>>>>>> afterwards) >>>>>>>> > https://github.com/apache/iceberg/pull/13070 >>>>>>>> > >>>>>>>> > Other items in the 1.10.0 milestone will probably have to be >>>>>>>> pushed to the next 1.11.0 release >>>>>>>> > https://github.com/apache/iceberg/milestone/54 >>>>>>>> > >>>>>>>> > Thanks, >>>>>>>> > Steven >>>>>>>> >>>>>>>
