AnandInguva commented on issue #28802: URL: https://github.com/apache/beam/issues/28802#issuecomment-1745678870
AsIter view_fn Iterable might look one element at a time and this could be more for the side input cache on the GCS bucket? - https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-10-03_13_02_55-13940213406894669659 - Pipeline scaled up to ~ 47 workers for a simple job. AsList view fn List materializes so we wouldn’t need too many reads from the side input cache at GCS bucket? - https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-10-03_13_03_00-15374614790418733789 - Pipeline completed as expected For AsIter with state_cache_size=100 mb, - https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-10-03_13_23_37-9771630946918171566 - With state cache enabled, this completed as expected since side input gets cached. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
