comphead commented on issue #3702:
URL: https://github.com/apache/iceberg-rust/issues/3702#issuecomment-6953883904

   > IIUC, I think datafusion-comet uses iceberg-rust for data read/write task? 
It still relies on java library for tx actions?
   
   yeah, we built the PoC to be as close to java implementation as possible, 
just replacing IO heavy parts to `iceberg-rs`, let me add the diagram that 
should clarify things.
   
   ### Spark
   
   ```mermaid
   flowchart TB
       entry(["CALL rewrite_data_files"])
   
       subgraph DRIVER [Driver]
           proc["RewriteDataFilesSparkAction"]
           plan["Step 1: Plan<br/>planFiles: 24,000 files"]
           binpack["Step 2: Bin-pack<br/>BinPackRewriteFilePlanner<br/>150 
groups of 160 files, one per day"]
           runner["Step 3: Rewrite 
groups<br/>SparkBinPackFileRewriteRunner<br/>one job per group, 5 at a time"]
           split["SparkStagedScan<br/>1 read task per group"]
           coord["FileRewriteCoordinator"]
           commit["Step 4: Commit<br/>RewriteFiles: remove 24,000, add 150"]
       end
   
       subgraph EXEC [Executor task]
           scan["Scan: Iceberg Parquet reader, JVM<br/>160 files in sequence, 
~38 s waiting"]
           c2r["ColumnarToRow"]
           write["Write: Iceberg Parquet writer, JVM<br/>1 file, ~13.7 MiB"]
       end
   
       entry --> proc --> plan --> binpack --> runner --> split
       split -->|"1 job, 1 task"| scan --> c2r --> write
       write -->|"new file"| coord -->|"after 150 groups"| commit
   ```
   
   ```mermaid
   sequenceDiagram
       participant D as Driver
       participant E as Executor task
       participant S as S3
       D->>E: run group job (160 files)
       loop 160 files, in sequence
           E->>S: read file (S3A)
           S-->>E: ~240 ms per file
       end
       E->>E: rows to Iceberg Parquet writer
       E->>S: write 1 file (~13.7 MiB)
       E-->>D: new file
       Note over D,S: ~44 s per group, 13% CPU
   ```
   
   ### comet-iceberg
   
   Same driver code. Comet's changes are highlighted.
   
   ```mermaid
   flowchart TB
       entry(["CALL rewrite_data_files"])
   
       subgraph DRIVER [Driver]
           proc["RewriteDataFilesSparkAction<br/>Comet compaction off"]
           plan["Step 1: Plan, as Spark"]
           binpack["Step 2: Bin-pack, as Spark"]
           runner["Step 3: Rewrite groups, as Spark<br/>one job per group, 5 at 
a time"]
           cplan["Comet plan rules<br/>scan becomes 
CometIcebergNativeScan<br/>native write declined"]
           coord["FileRewriteCoordinator"]
           commit["Step 4: Commit, as Spark"]
       end
   
       subgraph EXEC [Executor task]
           scan["Scan: CometIcebergNativeScan<br/>iceberg-rust, 4 files at a 
time, ~0.65 s"]
           c2r["CometColumnarToRow"]
           write["Write: Iceberg Parquet writer, JVM<br/>1 file, ~13.7 MiB"]
       end
   
       entry --> proc --> plan --> binpack --> runner --> cplan
       cplan -->|"1 job, 1 task"| scan --> c2r --> write
       write -->|"new file"| coord -->|"after 150 groups"| commit
   
       classDef comet fill:#ffe9b3,stroke:#d97706,color:#000
       class cplan,scan,c2r comet
   ```
   
   ```mermaid
   sequenceDiagram
       participant D as Driver
       participant E as Executor task
       participant N as Native scan
       participant S as S3
       D->>E: run group job (160 files, serialized)
       E->>N: start scan (JNI)
       loop 160 files, 4 at a time
           N->>S: range reads (opendal)
       end
       N-->>E: Arrow batches (~0.65 s)
       E->>E: rows to Iceberg Parquet writer
       E->>S: write 1 file (~13.7 MiB)
       E-->>D: new file
       Note over D,S: ~5 s per group, 90% CPU
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to