Is https://github.com/apache/iceberg/pull/12670/ considered introducing a
broken change for catalog in 1.9.0? Now the client can send a list of
snapshots to the server to remove in TableMetadata class while the server
with the old Iceberg library still checks to make sure only a single
snapshotId is sent? Seems now we need to upgrade the server to use Iceberg
1.9.x.
New Client:
private Builder rewriteSnapshotsInternal(Collection<Long> idsToRemove,
boolean suppress) {
List<Snapshot> retainedSnapshots =
Lists.newArrayListWithExpectedSize(snapshots.size() -
idsToRemove.size());
Set<Long> snapshotIdsToRemove = Sets.newHashSet();
for (Snapshot snapshot : snapshots) {
long snapshotId = snapshot.snapshotId();
if (idsToRemove.contains(snapshotId)) {
snapshotsById.remove(snapshotId);
if (!suppress) {
snapshotIdsToRemove.add(snapshotId);
}
}
}
if (!snapshotIdsToRemove.isEmpty()) {
* changes.add(new
MetadataUpdate.RemoveSnapshots(snapshotIdsToRemove));*
...
Old Server:
private static MetadataUpdate readRemoveSnapshots(JsonNode node) {
Set<Long> snapshotIds = JsonUtil.getLongSetOrNull(SNAPSHOT_IDS, node);
Preconditions.checkArgument(
snapshotIds != null && *snapshotIds.size() == 1*,
"Invalid set of snapshot ids to remove. Expected one value but
received: %s",
snapshotIds);
Long snapshotId = Iterables.getOnlyElement(snapshotIds);
return new MetadataUpdate.RemoveSnapshot(snapshotId);
}
On Fri, May 16, 2025 at 10:58 AM Fokko Driesprong <fo...@apache.org> wrote:
> Hey Russell,
>
> Thanks for driving this. Talking about making the security scanner happy:
> https://github.com/RussellSpitzer/iceberg/pull/5
>
> JB, let me know if you need any help with the Avro release!
>
> Kind regards,
> Fokko
>
> Op vr 16 mei 2025 om 19:56 schreef Driesprong, Fokko <fo...@driesprong.frl
> >:
>
>> Hey Russell,
>>
>> Thanks for driving this. Talking about making the security scanner happy:
>> https://github.com/RussellSpitzer/iceberg/pull/5
>>
>> JB, let me know if you need any help with the Avro release!
>>
>> Kind regards,
>> Fokko
>>
>> Op vr 16 mei 2025 om 19:39 schreef Russell Spitzer <
>> russell.spit...@gmail.com>:
>>
>>> Steven explained the Flink issue to me, Flink 2.0 isn't in 1.9.0 so not
>>> an issue.
>>>
>>> On Fri, May 16, 2025 at 12:20 PM Russell Spitzer <
>>> russell.spit...@gmail.com> wrote:
>>>
>>>> Ok so far of the lists proposed above I only picked 2 fixes that apply
>>>> cleanly and (we double checked)
>>>> actually apply to 1.9.0. Some of the fixes above need other commits
>>>> which aren't in 1.9.0 so aren't an
>>>> issue. If anyone else has any other issues let me know.
>>>>
>>>> The only one i'm not sure about is the Flink 2.0 Lock code, if someone
>>>> with flink expertise can ping me I
>>>> would appreciate it since I can't figure out how the patch applies to
>>>> 1.9.0.
>>>>
>>>> https://github.com/apache/iceberg/pull/13081
>>>>
>>>> On Tue, May 13, 2025 at 11:04 PM Jean-Baptiste Onofré <j...@nanthrax.net>
>>>> wrote:
>>>>
>>>>> The new Avro release will content security improvement (and only
>>>>> this). So even if not strictly required (as iceberg is not impacted), it
>>>>> would be interesting to have security scanner happy ;)
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>> Le mar. 13 mai 2025 à 23:24, Péter Váry <peter.vary.apa...@gmail.com>
>>>>> a écrit :
>>>>>
>>>>>> Do we really want to include a new lib version in a maintenance
>>>>>> release? In the past, we have seen issues when upgrading libs. Avro is
>>>>>> very
>>>>>> important, as it is used for metadata files. I would rather not include a
>>>>>> new version, unless it is absolutely necessary.
>>>>>>
>>>>>> On Tue, May 13, 2025, 06:42 Jean-Baptiste Onofré <j...@nanthrax.net>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> I did a fix/improvement on Avro. I will propose to do new Avro
>>>>>>> releases.
>>>>>>> Maybe worth to include in Iceberg 1.9.1 if the timing is ok.
>>>>>>>
>>>>>>> Regards
>>>>>>> JB
>>>>>>>
>>>>>>> Le lun. 12 mai 2025 à 20:03, Russell Spitzer <
>>>>>>> russell.spit...@gmail.com> a écrit :
>>>>>>>
>>>>>>>> I'd rather we didn't get any "feature" sorts of things in like
>>>>>>>> * Enable HTTP proxy support for the client used by REST Catalog
>>>>>>>> #12406 <https://github.com/apache/iceberg/pull/12406>
>>>>>>>> * GCP: Support multiple storage credential prefixes #12881
>>>>>>>> <https://github.com/apache/iceberg/pull/12881>
>>>>>>>>
>>>>>>>> These seem like refactors (no-ops for end users)
>>>>>>>> * Flink: Fix typo in JdbcLockFactory #12940
>>>>>>>> <https://github.com/apache/iceberg/pull/12940>
>>>>>>>> * Flink: Change Preconditions import from flink util to guava
>>>>>>>> #12939 <https://github.com/apache/iceberg/pull/12939>
>>>>>>>>
>>>>>>>>
>>>>>>>> I think these are real bugs we should fix :
>>>>>>>> * Core: Ensure reactivated view version uses correct timestamp
>>>>>>>> #12821 <https://github.com/apache/iceberg/pull/12821>
>>>>>>>> * Flink: Add lockFactory open in LockRemover for table maintenance
>>>>>>>> #12900 <https://github.com/apache/iceberg/pull/12900>
>>>>>>>>
>>>>>>>> Low Priority :
>>>>>>>> * Core: Broaden exception handling in writer clean up logic #12863
>>>>>>>> <https://github.com/apache/iceberg/pull/12863>
>>>>>>>> * Core: Disallow creation of invalid PartitionSpec #12887
>>>>>>>> <https://github.com/apache/iceberg/pull/12887>
>>>>>>>> * Core: Fix Kryo ser/de with StorageCredential config #12882
>>>>>>>> <https://github.com/apache/iceberg/pull/12882>
>>>>>>>> * Build, Core: Let RevAPI compare against 1.9.0 / Fix API breakage
>>>>>>>> around StorageCredential #12930
>>>>>>>> <https://github.com/apache/iceberg/pull/12930> // Is this a
>>>>>>>> dangerous time to change this? We are only doing a point release
>>>>>>>>
>>>>>>>>
>>>>>>>> The goal should be to just get in bug fixes for 1.9.0
>>>>>>>>
>>>>>>>> For me the priority goes
>>>>>>>> Highest -
>>>>>>>> Regressions - anything breaking the previous release (1.8.x)
>>>>>>>> Serious Bug Fixes - Correctness issues or major performance bugs
>>>>>>>> Minor Bug Fixes - Typos/ build things
>>>>>>>> --- Red Line
>>>>>>>> New Functionality / parameters ect
>>>>>>>> Lowest
>>>>>>>>
>>>>>>>> So I would avoid any "nice-to-have" items if we can and minimize
>>>>>>>> the changeset.
>>>>>>>>
>>>>>>>> On Mon, May 12, 2025 at 12:22 PM Yufei Gu <flyrain...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks Kevin for the list! That looks good to me.
>>>>>>>>> Looking forward to getting these fixes out!
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, May 12, 2025 at 10:19 AM Kevin Liu <kevinjq...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Russell,
>>>>>>>>>>
>>>>>>>>>> I went through the commits since 1.9.x release,
>>>>>>>>>> https://github.com/apache/iceberg/compare/1.9.x...main
>>>>>>>>>>
>>>>>>>>>> Here are some possible candidates for 1.9.1 patch release,
>>>>>>>>>> * Core: Fix Kryo ser/de with StorageCredential config #12882
>>>>>>>>>> <https://github.com/apache/iceberg/pull/12882>
>>>>>>>>>> * Core: Ensure reactivated view version uses correct timestamp
>>>>>>>>>> #12821 <https://github.com/apache/iceberg/pull/12821>
>>>>>>>>>> * Flink: Add lockFactory open in LockRemover for table
>>>>>>>>>> maintenance #12900 <https://github.com/apache/iceberg/pull/12900>
>>>>>>>>>> * Flink: Fix typo in JdbcLockFactory #12940
>>>>>>>>>> <https://github.com/apache/iceberg/pull/12940>
>>>>>>>>>> * Flink: Change Preconditions import from flink util to guava
>>>>>>>>>> #12939 <https://github.com/apache/iceberg/pull/12939>
>>>>>>>>>> * Core: Broaden exception handling in writer clean up logic
>>>>>>>>>> #12863 <https://github.com/apache/iceberg/pull/12863>
>>>>>>>>>> * Enable HTTP proxy support for the client used by REST Catalog
>>>>>>>>>> #12406 <https://github.com/apache/iceberg/pull/12406>
>>>>>>>>>> * Core: Disallow creation of invalid PartitionSpec #12887
>>>>>>>>>> <https://github.com/apache/iceberg/pull/12887>
>>>>>>>>>> * GCP: Support multiple storage credential prefixes #12881
>>>>>>>>>> <https://github.com/apache/iceberg/pull/12881>
>>>>>>>>>> * Build, Core: Let RevAPI compare against 1.9.0 / Fix API
>>>>>>>>>> breakage around StorageCredential #12930
>>>>>>>>>> <https://github.com/apache/iceberg/pull/12930>
>>>>>>>>>>
>>>>>>>>>> IMO most of these are "nice to have" as part of 1.9.1.
>>>>>>>>>> Let me know what you think!
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Kevin Liu
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, May 12, 2025 at 9:41 AM Russell Spitzer <
>>>>>>>>>> russell.spit...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I haven't gotten any other issues for 1.9.1 on the milestone and
>>>>>>>>>>> no one has responded here.
>>>>>>>>>>> I think it's important that we get a version of Iceberg out with
>>>>>>>>>>> a working Version function
>>>>>>>>>>> so I'll start a release today or tomorrow for a vote.
>>>>>>>>>>>
>>>>>>>>>>> On Sat, May 3, 2025 at 1:22 AM Jean-Baptiste Onofré <
>>>>>>>>>>> j...@nanthrax.net> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Russ
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, agree. Your PR is good and already merged.
>>>>>>>>>>>>
>>>>>>>>>>>> I don't have anything blocker for 1.9.1 (still working on
>>>>>>>>>>>> source-ids,
>>>>>>>>>>>> but definitely not for 1.9.1).
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks !
>>>>>>>>>>>> Regards
>>>>>>>>>>>> JB
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, May 2, 2025 at 11:47 PM Russell Spitzer
>>>>>>>>>>>> <russell.spit...@gmail.com> wrote:
>>>>>>>>>>>> >
>>>>>>>>>>>> > Hey y'all!
>>>>>>>>>>>> >
>>>>>>>>>>>> > Thanks to @suilis we have learned that IcebergBuild.version()
>>>>>>>>>>>> is returning unspecified for Iceberg 1.9.0. I have a PR up
>>>>>>>>>>>> > to fix this and I think this is a clear reason to do a 1.9.1
>>>>>>>>>>>> as soon as possible. I know we have a few other issues that
>>>>>>>>>>>> > may need to be fixed as well so let's make sure we get all
>>>>>>>>>>>> those listed and I can do a release when they are ready.
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> > Please respond if you have any concerns or you have any
>>>>>>>>>>>> issues that need to go into a 1.9.1,
>>>>>>>>>>>> > Russ
>>>>>>>>>>>>
>>>>>>>>>>>
