JacobSMoller commented on issue #6906: URL: https://github.com/apache/paimon/issues/6906#issuecomment-3847149898
@novakov-alexey after looking at this a bit more with @muttcg it does indeed look like the iceberg rest code will never really work - at least only if you add the iceberg rest settings at the very beginning of your tables lifecycle and then dont change the schema. Because paimons rest integration already is sort of hacky by having to add an empty schema to avoid some field id problems that Iceberg has an opinion about so they go from id: 0 -> 1 , 1->2, n->n+1 and then paimon does and update to get them back to 0, 1, 2 etc. after that paimon tracks the schema of iceberg rest as schema n+1, but if you do changes like alter table set options, paimon writes a new schema, which also triggers a new schema in iceberg - but in iceberg the paimon table options have no meaning and the actual fields haven't changed - iceberg sees this and deduplicates the schemas staying on n, but now paimon is on n+1 so when it next checks the rest catalog the n+1 schema trick no longer works. I think it is solvable but the whole iceberg commiter needs an overhaul as far as I can tell. @muttcg correct me if im totally of track :) https://polaris.apache.org/blog/2026/01/12/mapping-legacy-and-heterogeneous-datalakes-in-apache-polaris/ we are using Polaris and they have a notifications API, where you can send iceberg metadata and treat it as an external catalog so Polaris can only read it - but at least for our usecase that also makes sense because Paimon owns and writes the data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
