mbutrovich commented on PR #2156: URL: https://github.com/apache/iceberg-rust/pull/2156#issuecomment-3945272446
I'll take a look. In theory there's nothing stopping from generating FileScanTasks that span pieces of the files now (this is what Comet does with the existing reader). I've still been running into mismatches in parallelism though and trying to get more CPU utilization out of the Iceberg scan stages of Comet jobs, even when we've properly dispatched a bunch of I/O requests. I suspect you could be onto something here. Thanks! I'll take a pass this week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
