[GitHub] flink pull request: [FLINK-951] Reworking of Iteration Synchroniza...
Github user markus-h commented on the pull request: https://github.com/apache/flink/pull/570#issuecomment-214833775 Sorry for not driving this further. I think this pull request is now way too outdated to have any chance of rebasing it to the current master, therefore I will close it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-951] Reworking of Iteration Synchroniza...
Github user markus-h closed the pull request at: https://github.com/apache/flink/pull/570 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-951] Reworking of Iteration Synchroniza...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/570#issuecomment-100996059 I had a look at the pull request and I like very much what it tries to do. The problem right now is that I can hardly say without investing a lot of time whether this is in good shape to merge. This pull request does a at least two very big things at the same time: - Move iteration synchronization to the JobManager - Unify aggregators and accumulators into one. With all the example / testcase adjustments, this becomes a lot to review. The description of the pull request also does not make it easy, since many questions and decisions that arise are not explained: - What interface do the unified aggregators/accumulators follow: The aggregators, or the accumulators. - How is the blocking superstep synchronization currently done. With actor ask? - How is the aggregator/accumulator unification achieved, when aggregators are created per superstep, and accumulators once? This is a lot for a very delicate and critical mechanism. I think if we want to merge this, we would need more details on how things were changed (what is the concept behind the changed, not just what are the code diffs). We may need to break it into multiple self-contained changes that we can individually review and merge, to make sure that it gets properly checked and will work robustly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-951] Reworking of Iteration Synchroniza...
Github user uce commented on the pull request: https://github.com/apache/flink/pull/570#issuecomment-90865992 Have the latest commits fixed it? :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-951] Reworking of Iteration Synchroniza...
Github user markus-h commented on the pull request: https://github.com/apache/flink/pull/570#issuecomment-90694363 There seems to be a race condition somewhere in my code but I have trouble finding it since I can not reproduce it locally. I thought my last change would fix it but it didn't. So if somebody has some free time and knows a bit about race conditions feel free to help me :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-951] Reworking of Iteration Synchroniza...
Github user markus-h commented on the pull request: https://github.com/apache/flink/pull/570#issuecomment-89849730 Thanks for your comments! I will try to revert my formattings. I am used to doing STRG+F while programming that probably changed the formatting. I also got rid of the Thread.sleep(). The problem was acutally a different one. Akka delivered the same response object to threads on the same machine, but I thought it would be copies. Now I do a hard copy of the response, that seems to fix the problem. I am not sure how to test the interaction between IterationHead and JM though. Is there some similiar testcase that I could use as a basis? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-951] Reworking of Iteration Synchroniza...
GitHub user markus-h opened a pull request: https://github.com/apache/flink/pull/570 [FLINK-951] Reworking of Iteration Synchronization, Accumulators and Aggregators Iteration synchronization through JobManager Unification of Accumulators and Aggregators (removal of former Aggregators) Adjusted testcases accordingly I redid the work of my very old pull request https://github.com/apache/flink/pull/36 A more detailed description can be found in jira https://issues.apache.org/jira/browse/FLINK-951 I came across some unexpected behaviour with akka that made a small hack neccessary. Perhaps somebody with more experience in akka can find a better solution. See IterationHeadPactTask line 392. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markus-h/incubator-flink iterationsAndAccumulatorsRework2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/570.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #570 commit 5492487892ff99f10fccdb075404dedaa3371ff7 Author: Markus Holzemer markus.holze...@gmx.de Date: 2015-04-02T15:56:19Z Iteration synchronization through JobManager Unification of Accumulators and Aggregators (removal of former Aggregators) Adjusted testcases accordingly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
