1) Yes, that's right. After commit time we have the serialized chain of transactions.
2) Yes, it does. If you're rewriting whole data files then the validations used to verify that the result of the transaction is unchanged are a little different. For lazy updates (merge-on-read) Iceberg has to check for new conflicting delete files. For eager (copy-on-write), Iceberg will check that the updated files were not concurrently removed. Both need to check that there were no inserts that would have been affected by the transaction. 3.1) For snapshot isolation, you ignore concurrent modification if the commit can be applied. This is useful when the order doesn't really matter. For example, if you run `DELETE WHERE id = 5` and `INSERT (id, ...) VALUES (5, ...)` concurrently, then there is a perfectly valid ordering where the delete succeeds and then the insert attempts to commit. There's also a problematic ordering where the insert succeeds, but the delete fails because it should have deleted the new row. For cases like this, you can use `write.delete.isolation-level=snapshot` to avoid the failure, even when the delete happens second. 3.2) For eager updates, two concurrent processes would rewrite the same file. In that case, one would fail because it can't apply its changes (remove A1, add A2) in both isolation levels. For lazy updates, you'd end up with a difference. The two writers would create delete files, D1 and D2, both removing rows from A. Under serializable isolation, Iceberg would need to verify that there are no updates matching the conflict filter in D1 (verified using column ranges). For snapshot isolation you would not validate the conflict because you're ignoring changes after the transaction starts. On Thu, May 4, 2023 at 5:24 PM Nirav Patel <nira...@gmail.com> wrote: > Thanks Ryan/Szehon for responding and clarifying some concepts for me. I > got my answer but just to further clarify on transactions, (not urgent but > if you got time) > > 1) Isolation level is only check during commit. > 2) does cow or merge on read has any bearing on how this transaction > semantic works ? > 3) for multiple writers scenario does > > 1. for snapshot isolation: Ryan how does your example scenario play > differently if isolation level is snapshot? > 2. instead of adding two new files to a snapshot if both writers > update a same file or same record with non conflicting changes then how > these two isolation levels handles it differently at commit time? > > > > > On Thu, May 4, 2023 at 3:22 PM Szehon Ho <szehon...@apple.com.invalid> > wrote: > >> Whoops, I didn’t see Ryan answer already. >> >> On May 4, 2023, at 3:18 PM, Szehon Ho <szehon...@apple.com.INVALID> >> wrote: >> >> Hi, >> >> I believe it only matters if you have conflicting commits. For single >> writer case, I think you are right and it should not matter, so you may >> save very slightly in performance by turning it to Snapshot Isolation. The >> checks are metadata checks though, so I would think it will not be a >> signfiicant performance difference. >> >> In general, the isolation levels in Iceberg work by checking before >> commit to see if there are any conflicting changes to data files about to >> be committed, from when the operation first started (ie, starting snapshot >> id). So if there is a failure due to the isolation level, I believe the >> error bubbles back the application to try again, hence ‘pessimistic’. >> >> Note, metadata conflicts are automatically retried and should rarely >> bubble up to user, so only in case of data isolation level conflict (ie, >> you delete a file that is currently being rewritten by another operation), >> will error-handling be required. >> >> Hope that helps >> Szehon >> >> On May 4, 2023, at 12:19 PM, Nirav Patel <nira...@gmail.com> wrote: >> >> I am trying to ingest data into iceberg table using spark streaming. >> There are no multiple writers to same data at the moment. According to >> iceberg >> api >> <https://iceberg.apache.org/javadoc/0.11.0/org/apache/iceberg/IsolationLevel.html#:%7E:text=Both%20of%20them%20provide%20a,environments%20with%20many%20concurrent%20writers.> >> default >> isolation level for table is serializable . I want to understand if >> there is only a single application (single spark streaming job in my case) >> writing to iceberg table is there any advantage or disadvantage over using >> serializable or a snapshot isolation ? Is there any performance impact >> of using serializable when only one application is writing to table? >> Also it seems iceberg allows all writers to write into snapshot and use OCC >> to decide if one needs to retry because it was late. In this case how it is >> serializable at all? isn't serilizability achieved via >> pessimistic concurrency control? Would like to understand how iceberg >> implement serializable isolation level and how it is different than >> snapshot isolation ? >> >> Thanks >> >> >> >> -- Ryan Blue Tabular
