[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14727372#comment-14727372 ] ASF GitHub Bot commented on FLINK-2448: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1031 > registerCacheFile fails with MultipleProgramsTestbase > - > > Key: FLINK-2448 > URL: https://issues.apache.org/jira/browse/FLINK-2448 > Project: Flink > Issue Type: Bug > Components: Tests >Reporter: Chesnay Schepler >Assignee: Sachin Goel >Priority: Minor > Fix For: 0.10 > > > When trying to register a file using a constant name an expection is thrown > saying the file was already cached. > This is probably because the same environment is reused, and the cacheFile > entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14725162#comment-14725162 ] ASF GitHub Bot commented on FLINK-2448: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-136669559 I think this looks good. Will merge this... > registerCacheFile fails with MultipleProgramsTestbase > - > > Key: FLINK-2448 > URL: https://issues.apache.org/jira/browse/FLINK-2448 > Project: Flink > Issue Type: Bug > Components: Tests >Reporter: Chesnay Schepler >Assignee: Sachin Goel >Priority: Minor > > When trying to register a file using a constant name an expection is thrown > saying the file was already cached. > This is probably because the same environment is reused, and the cacheFile > entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14725643#comment-14725643 ] ASF GitHub Bot commented on FLINK-2448: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-136780181 Okay. Great. :) > registerCacheFile fails with MultipleProgramsTestbase > - > > Key: FLINK-2448 > URL: https://issues.apache.org/jira/browse/FLINK-2448 > Project: Flink > Issue Type: Bug > Components: Tests >Reporter: Chesnay Schepler >Assignee: Sachin Goel >Priority: Minor > > When trying to register a file using a constant name an expection is thrown > saying the file was already cached. > This is probably because the same environment is reused, and the cacheFile > entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723326#comment-14723326 ] ASF GitHub Bot commented on FLINK-2448: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-136349300 @StephanEwen, this should be mergeable now. I've separated the Gelly change as a second commit. > registerCacheFile fails with MultipleProgramsTestbase > - > > Key: FLINK-2448 > URL: https://issues.apache.org/jira/browse/FLINK-2448 > Project: Flink > Issue Type: Bug > Components: Tests >Reporter: Chesnay Schepler >Assignee: Sachin Goel >Priority: Minor > > When trying to register a file using a constant name an expection is thrown > saying the file was already cached. > This is probably because the same environment is reused, and the cacheFile > entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709380#comment-14709380 ] ASF GitHub Bot commented on FLINK-2448: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-134227222 I do not quite understand the idea behind this pull request. It adds a public `setAsShared()` method to the core `ExecutionEnvironment` which is only relevant for tests. Can you not simply change the test base such that the TestContextFactory creates a new Environment every time and remembers that as the last environment? Do some tests assume shared environments? registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Assignee: Sachin Goel Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709515#comment-14709515 ] ASF GitHub Bot commented on FLINK-2448: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-134265716 Yes, that's perfectly doable. I actually did that initially. The only reason to add it to the `ExecutionEnvironment` was to remove that little bit of code duplication in `getLastJobExecutionResult`. I guess we should keep it restricted to test environments only. Apologies. registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Assignee: Sachin Goel Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708305#comment-14708305 ] ASF GitHub Bot commented on FLINK-2448: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-133788218 Travis passes. :) I asked on the dev list about the two small changes in flink-gelly. @vasia registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706523#comment-14706523 ] ASF GitHub Bot commented on FLINK-2448: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-133365847 So, this is now removing some safeguard checks in the test to make this work. That does not seem quite right. Why not keep the environment and keep access to the latest execution result? registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706531#comment-14706531 ] ASF GitHub Bot commented on FLINK-2448: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-133370663 To access the last execution result, the environment needs to be shared. And since we're creating a new execution environment in the program, it wouldn't work. I'm not sure if the assert for last execution result not being null is useful. If the program succeeded, shouldn't this be always *not null*? registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706536#comment-14706536 ] ASF GitHub Bot commented on FLINK-2448: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-133371332 Yes, if you read the message of the checks you removes, it is precisely to figure that out: Whether a program actually triggered execution. We've build a kick-ass distributed system here. I am quite sure we can a reference to some field into another class, if we want... registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706812#comment-14706812 ] ASF GitHub Bot commented on FLINK-2448: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-133449024 Okay, I figured out a way to share the environments. :) Let me know if this is a valid approach. Waiting for travis to pass. registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704428#comment-14704428 ] ASF GitHub Bot commented on FLINK-2448: --- Github user zentol commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-132917505 using clearsinks would cause the above example with collect() to fail, since this clears sinks aswell. (the sinks are only not cleared when getExecutionPlan is called, it has nothing to do with a new execution). just checking the filepath wouldn't work in my case, i distribute a folder that resides in the same location whose content varies. registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704479#comment-14704479 ] ASF GitHub Bot commented on FLINK-2448: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-132927894 Ah yes, you're right. We cannot use `clearSinks`. To answer the second part, the folder will still be added. It is only at runtime that all the directories and files inside a folder are recursively copied. So, just adding the folder once should work. registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704506#comment-14704506 ] ASF GitHub Bot commented on FLINK-2448: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-132934917 I think the reason to not create a new environment every time was simply that it was easier to write that way and it never hurt before. You can run multiple programs on the same environment after all. For tests, it is cleaner, though, to have a fresh one, as otherwise you may inherit configuration from a previous test (as is the case with the registered cache files). registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704606#comment-14704606 ] ASF GitHub Bot commented on FLINK-2448: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-132958065 Okay. For now, I have modified all Test environments which were using a context factory, except for the `PreviewPlanEnvironment` which cannot be done as the test depends on the environment being shared. registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704737#comment-14704737 ] ASF GitHub Bot commented on FLINK-2448: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-132987393 It turns out the `JavaProgramTestBase` accesses the shared environment's execution result. However, it is used nowhere. I have removed the unused code and modified all Test environment factories to always return a new Environment. registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14703405#comment-14703405 ] ASF GitHub Bot commented on FLINK-2448: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-132704210 A call to clear cache files does not hurt, but let's fix the MultipleProgramsTest base to produce fresh ExecutionEnvironments in its factory, rather than repeatedly handing out the same one. That is a good fix anyways. registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704229#comment-14704229 ] ASF GitHub Bot commented on FLINK-2448: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-132871675 Was there a specific reason to not generate fresh environments in the first place? Or was it just to minimize object creations? Here's what I propose though: Since there is already a `clearSinks` argument to `createProgramPlan`, we can use that as an indicator that the cache file list has to be cleared too, since it indicates a new Execution. Further, instead of throwing an exception in case a `name` is re-registered, we check if the `filePath` is actually the same. If not, then we throw an exception, otherwise just silently ignore it. registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14700983#comment-14700983 ] ASF GitHub Bot commented on FLINK-2448: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-132136444 I am not sure this is solving it the right way. I find it more intuitive if cache files stay registered. They are part of the environment and should be sticky, like configuration settings as well (they don't get lost after execution). We could add a method to clear registered cache files. But if the problem is really only the MultiplProgramsTest base, then this should be fixed to return a new ExecutionEnvironment from a (the) TestEnvironmentFactory every time you call `getExecutionEnvironment()`. registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701000#comment-14701000 ] ASF GitHub Bot commented on FLINK-2448: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-132140770 I'm not sure if I can form an opinion on this. I personally haven't used them at all with Flink, only with Hadoop. A call to clear cache files certainly makes sense though. registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699254#comment-14699254 ] Sachin Goel commented on FLINK-2448: I'm not sure what you mean by running multiple programs in a single method. Do you mean, for example, running several jobs in one test? In that case too, as long as the environment is re-created, it should not fail. registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699295#comment-14699295 ] Sachin Goel commented on FLINK-2448: Aha. It makes sense. The `TestEnvironment` context factory returns the same object always. The second `getEnvironment` call doesn't actually do anything. registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699353#comment-14699353 ] ASF GitHub Bot commented on FLINK-2448: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-131772983 Ah yes. You're right. Lemme see if I can write a workaround to this. registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699373#comment-14699373 ] ASF GitHub Bot commented on FLINK-2448: --- Github user sachingoel0101 commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-131781343 Should there be a unit test to verify this functionality? registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699318#comment-14699318 ] ASF GitHub Bot commented on FLINK-2448: --- GitHub user sachingoel0101 opened a pull request: https://github.com/apache/flink/pull/1031 [FLINK-2448]Clear cache file list in ExecutionEnvironment after program plan creation You can merge this pull request into a Git repository by running: $ git pull https://github.com/sachingoel0101/flink master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1031.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1031 commit 199fa54cec1c6a9749fbbebf009267a997a7b275 Author: Sachin Goel sachingoel0...@gmail.com Date: 2015-08-17T10:00:52Z [FLINK-2448][hotfix]Clear cache file list in ExecutionEnvironment after registering with Plan registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[
https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699275#comment-14699275
]
Chesnay Schepler commented on FLINK-2448:
-
yep that's what i mean. the environment is retrieved using
ExecutionEnvironment.getEnvironment() for both jobs.
essentially this is what runs:
{code}
@Test
public void MyTest() throws Exception {
ExecutionEnvironment env = ExecutionEnvironment.getEnvironment();
env.registerCacheFile(X)
doSomeStuff
env.execute()
env = ExecutionEnvironment.getEnvironment();
env.registerCacheFile(X)
doSomeStuff
env.execute()
}
{code}
registerCacheFile fails with MultipleProgramsTestbase
-
Key: FLINK-2448
URL: https://issues.apache.org/jira/browse/FLINK-2448
Project: Flink
Issue Type: Bug
Components: Tests
Reporter: Chesnay Schepler
Priority: Minor
When trying to register a file using a constant name an expection is thrown
saying the file was already cached.
This is probably because the same environment is reused, and the cacheFile
entries are not cleared between runs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699345#comment-14699345 ] ASF GitHub Bot commented on FLINK-2448: --- Github user zentol commented on the pull request: https://github.com/apache/flink/pull/1031#issuecomment-131769530 Does this work properly with collect() calls? as in, would the following plan still work? `env = .. env.registerCacheFile() ... someSet.collect() doSomethingThatUsesTheCacheFile env.execute() ` if we wipe all cache entries in the collect() call, the files will not be registered in the execute(), right? The plans these methods create are separate i think. registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699163#comment-14699163 ] Chesnay Schepler commented on FLINK-2448: - this has got nothing to do with the Collection Environment, so I assume it does. registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[
https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699176#comment-14699176
]
Sachin Goel commented on FLINK-2448:
Could you share the failing code? I am using the {{MultipleProgramsTestBase}}
in the unit test for verifying the new Distributed Cache functionality and it
doesn't cause any problems.
registerCacheFile fails with MultipleProgramsTestbase
-
Key: FLINK-2448
URL: https://issues.apache.org/jira/browse/FLINK-2448
Project: Flink
Issue Type: Bug
Components: Tests
Reporter: Chesnay Schepler
Priority: Minor
When trying to register a file using a constant name an expection is thrown
saying the file was already cached.
This is probably because the same environment is reused, and the cacheFile
entries are not cleared between runs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699204#comment-14699204 ] Chesnay Schepler commented on FLINK-2448: - alright, I'll check again, will be back shortly :) registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2448) registerCacheFile fails with MultipleProgramsTestbase
[ https://issues.apache.org/jira/browse/FLINK-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699009#comment-14699009 ] Sachin Goel commented on FLINK-2448: [~Zentol], could you check this again on the latest master? registerCacheFile fails with MultipleProgramsTestbase - Key: FLINK-2448 URL: https://issues.apache.org/jira/browse/FLINK-2448 Project: Flink Issue Type: Bug Components: Tests Reporter: Chesnay Schepler Priority: Minor When trying to register a file using a constant name an expection is thrown saying the file was already cached. This is probably because the same environment is reused, and the cacheFile entries are not cleared between runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
