ErikBPF opened a new issue, #6103: URL: https://github.com/apache/datafusion-comet/issues/6103
## Problem The Iceberg core tests are sharded by a task-agnostic class-name predicate, but `iceberg-spark-extensions` runs unsharded (`iceberg_spark_test_reusable.yml:221-227`) and measured **55m24s** in queue run `35022044196` — the single longest job in that run and the largest ownable saving. ## Change - Give the extensions job the same shard matrix: pass `-PcometShardIndex` / `-PcometShardCount` and `-PcometShardTask` for the extensions task, with a task-specific artifact name. - Add an extensions coverage check alongside the existing core one. ## Expected gain 55m24s unsharded → ~15-20 min ⇒ ~35 queue-min/run × ~38 runs ≈ **1,300 runner-min/72 h (~2 %)**. The largest single ownable saving. ## Constraint `dev/ci/check-iceberg-shards.py:43` hard-codes `SHARD_COUNT = 4` and asserts every manifest's count equals it, so extensions must use count 4 (or the constant must become per-task). The Gradle predicate is a pure hash of the class name and is task-agnostic (`dev/ci/iceberg-test-shards.gradle`). ## Verification - One label run produces balanced per-shard candidate counts (extensions balance is unverified until then) and the coverage check passes: the union of shard candidates equals the unsharded inventory. - Queue wall clock for the extensions job 55m24s → target ~15-20 min. ## Risk / rollback Shard imbalance could make one shard as slow as the unsharded job; the coverage check prevents dropped tests. Revert the workflow and the matrix emitter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
