CTTY opened a new issue, #108:
URL: https://github.com/apache/datafusion-iceberg/issues/108

   ## Problem
   
   #67 plans `FileScanTask`s before execution and distributes them across
   `target_partitions`. Each task is a whole file: `collect_into` in
   `read/file_planning.rs` round-robins tasks as `plan_files()` yields them, 
with
   `start`/`length` covering the full file.
   
   A table with fewer large files than partitions leaves partitions idle, and a
   file with many row groups is read end to end by one partition. Files written
   by Spark or Flink are commonly 128 MB to 1 GB with many row groups, so this 
is
   the usual case.
   
   The reader already handles byte ranges: when `task.start() != 0 ||
   task.length() != 0`, `ArrowReader` keeps only the row groups starting inside
   `[start, start + length)` (`filter_row_groups_by_byte_range`). The missing
   piece was knowing the row-group boundaries at planning time.
   apache/iceberg-rust#3704 adds `FileScanTask::split_offsets()`, populated from
   `DataFile.split_offsets` (spec field 132), so a planner can cut at those
   offsets without opening the file.
   
   Cutting at arbitrary byte positions is not an option: a cut inside a row 
group
   makes two tasks open the file while one of them yields nothing.
   
   ## Proposal
   
   In eager planning, after `plan_files()` and before partition assignment:
   
   1. Pick a target split size: a session setting such as
      `iceberg.target_split_size_bytes` (Iceberg Java's `read.split.target-size`
      defaults to 128 MB), or total selected bytes / `target_partitions` with a
      floor.
   2. For each task with `length()` above the target and non-empty
      `split_offsets()`, cut it at offsets chosen to keep each piece near the
      target. Each piece is a clone of the task with its own `start`/`length`;
      delete files and the residual predicate stay on every piece.
   3. Tasks with no offsets, or below the target, stay whole.
   4. Assign the resulting tasks to partitions. Round-robin is enough for a 
first
      version; largest-first byte balancing as in
      datafusion-contrib/datafusion-distributed#788 is a natural follow-up.
   
   Expose the splitting step as a public function on the task list so
   `with_planned_files(groups)` consumers (DFD, Ballista, Comet) can reuse it.
   
   Lazy planning is out of scope; #84 covers that path.
   
   ## Open questions
   
   - Position deletes: each piece loads the same delete vectors. #67 shares the
     delete loader across partitions within an execution, so this should be one
     load per file rather than per piece; worth confirming.
   - Whether `EXPLAIN` should show the split count next to the file count.
   
   ## Dependencies
   
   - #67 merged.
   - iceberg-rust revision bumped past apache/iceberg-rust#3704.
   
   ## Validation
   
   - One file with 4 row groups and `target_partitions=4` yields 4 tasks, each
     partition reads exactly one row group, and results match the
     single-partition scan, including with position deletes and a predicate.
   - A file with `split_offsets` `None` or empty stays one task.
   - A file below the target stays one task regardless of offsets.
   - `insta` snapshot of the plan showing the split count.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to