comphead commented on issue #3702:
URL: https://github.com/apache/iceberg-rust/issues/3702#issuecomment-6953883904
> IIUC, I think datafusion-comet uses iceberg-rust for data read/write task?
It still relies on java library for tx actions?
yeah, we built the PoC to be as close to java implementation as possible,
just replacing IO heavy parts to `iceberg-rs`, let me add the diagram that
should clarify things.
### Spark
```mermaid
flowchart TB
entry(["CALL rewrite_data_files"])
subgraph DRIVER [Driver]
proc["RewriteDataFilesSparkAction"]
plan["Step 1: Plan<br/>planFiles: 24,000 files"]
binpack["Step 2: Bin-pack<br/>BinPackRewriteFilePlanner<br/>150
groups of 160 files, one per day"]
runner["Step 3: Rewrite
groups<br/>SparkBinPackFileRewriteRunner<br/>one job per group, 5 at a time"]
split["SparkStagedScan<br/>1 read task per group"]
coord["FileRewriteCoordinator"]
commit["Step 4: Commit<br/>RewriteFiles: remove 24,000, add 150"]
end
subgraph EXEC [Executor task]
scan["Scan: Iceberg Parquet reader, JVM<br/>160 files in sequence,
~38 s waiting"]
c2r["ColumnarToRow"]
write["Write: Iceberg Parquet writer, JVM<br/>1 file, ~13.7 MiB"]
end
entry --> proc --> plan --> binpack --> runner --> split
split -->|"1 job, 1 task"| scan --> c2r --> write
write -->|"new file"| coord -->|"after 150 groups"| commit
```
```mermaid
sequenceDiagram
participant D as Driver
participant E as Executor task
participant S as S3
D->>E: run group job (160 files)
loop 160 files, in sequence
E->>S: read file (S3A)
S-->>E: ~240 ms per file
end
E->>E: rows to Iceberg Parquet writer
E->>S: write 1 file (~13.7 MiB)
E-->>D: new file
Note over D,S: ~44 s per group, 13% CPU
```
### comet-iceberg
Same driver code. Comet's changes are highlighted.
```mermaid
flowchart TB
entry(["CALL rewrite_data_files"])
subgraph DRIVER [Driver]
proc["RewriteDataFilesSparkAction<br/>Comet compaction off"]
plan["Step 1: Plan, as Spark"]
binpack["Step 2: Bin-pack, as Spark"]
runner["Step 3: Rewrite groups, as Spark<br/>one job per group, 5 at
a time"]
cplan["Comet plan rules<br/>scan becomes
CometIcebergNativeScan<br/>native write declined"]
coord["FileRewriteCoordinator"]
commit["Step 4: Commit, as Spark"]
end
subgraph EXEC [Executor task]
scan["Scan: CometIcebergNativeScan<br/>iceberg-rust, 4 files at a
time, ~0.65 s"]
c2r["CometColumnarToRow"]
write["Write: Iceberg Parquet writer, JVM<br/>1 file, ~13.7 MiB"]
end
entry --> proc --> plan --> binpack --> runner --> cplan
cplan -->|"1 job, 1 task"| scan --> c2r --> write
write -->|"new file"| coord -->|"after 150 groups"| commit
classDef comet fill:#ffe9b3,stroke:#d97706,color:#000
class cplan,scan,c2r comet
```
```mermaid
sequenceDiagram
participant D as Driver
participant E as Executor task
participant N as Native scan
participant S as S3
D->>E: run group job (160 files, serialized)
E->>N: start scan (JNI)
loop 160 files, 4 at a time
N->>S: range reads (opendal)
end
N-->>E: Arrow batches (~0.65 s)
E->>E: rows to Iceberg Parquet writer
E->>S: write 1 file (~13.7 MiB)
E-->>D: new file
Note over D,S: ~5 s per group, 90% CPU
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]