rdsr opened a new pull request #111: Split files when planning scan tasks URL: https://github.com/apache/incubator-iceberg/pull/111 This addresses #37 by splitting files at target split size before bin packing. This does break the tests under ```com.netflix.iceberg.spark.source.TestFilteredScan``` which assume a split per partition,. I'm currently unclear how these tests should be handled. Should we keep the assumption around the number of tasks created or should we replace them with something else? If we can decide on how we'll handle the assumption around number of tasks, I can add tests to test out this feature.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
