[ https://issues.apache.org/jira/browse/BEAM-5708?focusedWorklogId=153977&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153977 ]
ASF GitHub Bot logged work on BEAM-5708: ---------------------------------------- Author: ASF GitHub Bot Created on: 12/Oct/18 18:29 Start Date: 12/Oct/18 18:29 Worklog Time Spent: 10m Work Description: angoenka commented on a change in pull request #6638: [BEAM-5708] Cache environment in portable flink runner URL: https://github.com/apache/beam/pull/6638#discussion_r224875048 ########## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java ########## @@ -104,9 +108,21 @@ private void scheduleRelease(JobInfo jobInfo) { WrappedContext wrapper = getCache().get(jobInfo.jobId()); Preconditions.checkState( wrapper != null, "Releasing context for unknown job: " + jobInfo.jobId()); - // Do not release this asynchronously, as the releasing could fail due to the classloader not being - // available anymore after the tasks have been removed from the execution engine. - release(wrapper); + + PipelineOptions pipelineOptions = + PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions()); + int environmentCacheTTLMillis = + pipelineOptions.as(PortablePipelineOptions.class).getEnvironmentCacheMillis(); + if (environmentCacheTTLMillis > 0) { + // Schedule task to clean the container later. + // Ensure that this class is loaded in the parent Flink classloader. + getExecutor() + .schedule(() -> release(wrapper), environmentCacheTTLMillis, TimeUnit.MILLISECONDS); Review comment: @tweise Makes sense. I will add that check. In addition of falling back to immediate release, I will also log a warning. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 153977) Time Spent: 1.5h (was: 1h 20m) > Support caching of SDKHarness environments in flink > --------------------------------------------------- > > Key: BEAM-5708 > URL: https://issues.apache.org/jira/browse/BEAM-5708 > Project: Beam > Issue Type: Improvement > Components: runner-flink > Reporter: Ankur Goenka > Assignee: Ankur Goenka > Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > Cache and reuse environment to improve performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)