reuvenlax opened a new pull request, #40321: URL: https://github.com/apache/beam/pull/40321
TestDataflowRunner previously set tempLocation to <tempRoot>/<jobName>/output/results, causing DataflowPipelineOptions.StagingLocationFactory` to derive a unique path for every single test job. Consequently, every test re-hashed ~300+ MB of classpath jars and re-queried/uploaded them to a brand-new GCS staging directory instead of reusing staged artifacts across tests. **Fix**: Defaults stagingLocation to a shared /staging/ directory across jobs in the same test run (while keeping tempLocation isolated per job at <tempRoot>/<jobName>/output/results). Caches SHA-256 file hashes and directory zip artifacts in memory keyed by (path, size, lastModifiedMillis) so unmodified classpath jars are hashed once per worker JVM instead of once per test. Caches resolved DataflowPackage lists per `stagingLocation, filesToStage) when running under TestDataflowRunner so each worker JVM stages/verifies the classpath once. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
