alexanderbianchi opened a new pull request, #107:
URL: https://github.com/apache/datafusion-iceberg/pull/107
## Summary
Allow lazy Iceberg scans to consume native `FileScanTask` streams supplied
by an application, while keeping file reading in `iceberg-datafusion`.
- Introduce `FileScanTaskSource` and the default, single-partition
`TableScanTaskSource`.
- Make `IcebergDataSource<T = TableScanTaskSource>` generic over task
delivery, preserving existing provider usage and the synchronous builder.
- Add `from_task_source()` for construction without a table/catalog and
`with_task_source()` for replacing discovery while preserving schema, reader
selection and fetch.
- Keep execution-time batch-size/page-index defaults for ordinary table
scans; accept explicitly configured readers without overriding their settings.
Explicit-reader clones share loaded delete files.
- Enforce partition bounds and apply per-partition fetch through
`LimitStream`, releasing task delivery when the limit is reached. Fetch zero
opens no task stream.
- Add eight focused tests and correct the existing schema/stream test to
execute partition `0` of its single-partition source.
## Scope
This is a standalone, main-based change, independent of #67 and eager file
planning. There is no worker scheduling, transport, serialization, eager
grouping, statistics computation, runtime-override API, provider hook or SQL
snapshot option in this PR. Statistics remain unknown. The task-source
interface does not promise bounded buffering or strict backpressure.
## Validation
Validated with the repository-pinned Rust **1.98.1**, without
dependency-version or lockfile changes:
- **29 focused tests passed:**
- 8 new `file_task_source` tests;
- 16 existing tests across `batch_size`, `page_index`, `limit_pushdown`,
`projection_pushdown` and `filter_pushdown`;
- 5 selected scan integration tests: `test_provider_plan_stream_schema`,
`test_table_projection`, `test_table_predict_pushdown`,
`test_plan_nodes_are_inspectable` and
`test_scan_after_schema_evolution_reads_provider_columns`.
- Scoped Clippy passed with `-D warnings` for the library and both changed
integration-test targets.
- Pinned rustfmt and whitespace checks passed.
- A negative check disabled reader fetch enforcement: only
`reaching_fetch_stops_before_requesting_another_task` was run and failed on
continued task consumption as intended. The implementation was restored
byte-for-byte and all 29 focused tests were rerun successfully.
The full workspace/SQL-logic suites and benchmarks were not run. Local
builds used debug-info level 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
