Re: Picking up Giraph 1.1.0 release
Hi! good news everyone! We've got just one JIRA left that is assigned to be fixed in 1.1: https://issues.apache.org/jira/browse/GIRAPH-747 Everything else is either resolved or just tracks the release activities. Based on this, I have create a release-1.1 branch and will proceed with Bigtop-based testing for the next week. Based on the result of system testing hopefully we would be able to release in a week or so. For the next couple of weeks please be extra careful marking JIRAs with 1.1 to get things into the release. I plan to announce our first RC mid-week. It would be really nice if folks can jump on it and provide some additional testing. Finally, if somebody can look at GIRAPH-747 that also would be very much appreciated! Thanks, Roman.
[jira] [Commented] (GIRAPH-886) Partition Balancing does not work
[ https://issues.apache.org/jira/browse/GIRAPH-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021757#comment-14021757 ] Lukas Nalezenec commented on GIRAPH-886: Hi Pavan, Thanks for your review. I am really busy this month. I wanted to refactor the first patch but it i havent finished it yet. I will make some simplified version. Partition Balancing does not work - Key: GIRAPH-886 URL: https://issues.apache.org/jira/browse/GIRAPH-886 Project: Giraph Issue Type: Bug Components: graph Affects Versions: 1.1.0 Reporter: Lukas Nalezenec Labels: patch Attachments: GIRAPH-886.patch PartitionBalancer returns PartitionOwners in list in arbitrary order but both Range and Hash partitioners are assuming that list is sorted by partitionId. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (GIRAPH-914) giraph-core compiled with cdh4.1.2 profile produces 50 MB fat jar.
Lukas Nalezenec created GIRAPH-914: -- Summary: giraph-core compiled with cdh4.1.2 profile produces 50 MB fat jar. Key: GIRAPH-914 URL: https://issues.apache.org/jira/browse/GIRAPH-914 Project: Giraph Issue Type: Bug Components: build Affects Versions: 1.1.0 Reporter: Lukas Nalezenec Priority: Critical When you compile giraph-core with cdh4.1.2 it has got 50 MB. Its too much. It has got lot of useless dependencies. I think that we can save a lot setting scope to provided. We could also make jython dependencies optional - jython dependencies has got 8 MB. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (GIRAPH-914) giraph-core compiled with cdh4.1.2 profile produces 50 MB fat jar.
[ https://issues.apache.org/jira/browse/GIRAPH-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Nalezenec updated GIRAPH-914: --- Attachment: effective.pom.xml dependency.tree giraph-core compiled with cdh4.1.2 profile produces 50 MB fat jar. -- Key: GIRAPH-914 URL: https://issues.apache.org/jira/browse/GIRAPH-914 Project: Giraph Issue Type: Bug Components: build Affects Versions: 1.1.0 Reporter: Lukas Nalezenec Priority: Critical Attachments: dependency.tree, effective.pom.xml When you compile giraph-core with cdh4.1.2 it has got 50 MB. Its too much. It has got lot of useless dependencies. I think that we can save a lot setting scope to provided. We could also make jython dependencies optional - jython dependencies has got 8 MB. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (GIRAPH-914) giraph-core compiled with cdh4.1.2 profile produces 50 MB fat jar.
[ https://issues.apache.org/jira/browse/GIRAPH-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Nalezenec updated GIRAPH-914: --- Priority: Major (was: Critical) giraph-core compiled with cdh4.1.2 profile produces 50 MB fat jar. -- Key: GIRAPH-914 URL: https://issues.apache.org/jira/browse/GIRAPH-914 Project: Giraph Issue Type: Bug Components: build Affects Versions: 1.1.0 Reporter: Lukas Nalezenec Attachments: dependency.tree, effective.pom.xml When you compile giraph-core with cdh4.1.2 it has got 50 MB. Its too much. It has got lot of useless dependencies. I think that we can save a lot setting scope to provided. We could also make jython dependencies optional - jython dependencies has got 8 MB. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Picking up Giraph 1.1.0 release
Actually, I was going to commit 874. Shall I commit it to release-1.1? On Mon, Jun 9, 2014 at 8:30 AM, Roman Shaposhnik ro...@shaposhnik.org wrote: Hi! good news everyone! We've got just one JIRA left that is assigned to be fixed in 1.1: https://issues.apache.org/jira/browse/GIRAPH-747 Everything else is either resolved or just tracks the release activities. Based on this, I have create a release-1.1 branch and will proceed with Bigtop-based testing for the next week. Based on the result of system testing hopefully we would be able to release in a week or so. For the next couple of weeks please be extra careful marking JIRAs with 1.1 to get things into the release. I plan to announce our first RC mid-week. It would be really nice if folks can jump on it and provide some additional testing. Finally, if somebody can look at GIRAPH-747 that also would be very much appreciated! Thanks, Roman. -- Claudio Martella
Re: Picking up Giraph 1.1.0 release
On Mon, Jun 9, 2014 at 2:14 PM, Claudio Martella claudio.marte...@gmail.com wrote: Actually, I was going to commit 874. Shall I commit it to release-1.1? Just keep committing to trunk (the usual way), but every time you think the JIRA should also go into 1.1 -- please ping me. Thanks, Roman.
Re: Review Request 21987: Detect crashes of Netty threads
On May 29, 2014, 6:40 p.m., Pavan Kumar Athivarapu wrote: findbugs-exclude.xml, line 46 https://reviews.apache.org/r/21987/diff/1/?file=597788#file597788line46 why are u removing these stuff, wouldn't it be fine to just add GraphTaskManager to allow exit I just removed unused code from GiraphYarnTask and hence it doesn't have System.exit() anymore and that made this check obsolete. On May 29, 2014, 6:40 p.m., Pavan Kumar Athivarapu wrote: giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/ExceptionHandler.java, line 29 https://reviews.apache.org/r/21987/diff/1/?file=597792#file597792line29 this catches exceptions only for inbound conenctions, do u want to do this for outbound as well. in that case would it not be better to just piggy back on the counters override exceptioncaught in those methods? I am fine with creating new classes or just overriding the method in counters Pavan Kumar Athivarapu wrote: one more thing (for exception handling) - inbound handler should be at the end and outbound handler should be at the beginning of the pipeline so u catch exceptions at the end of processing input - 1 u catch exceptions at the end of sending output - 2 I felt like it's nice to have a separate class for exception handling as it outlines it's intention and makes it clear how and when to use it. - Sergey --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21987/#review44270 --- On June 5, 2014, 10:26 p.m., Sergey Edunov wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21987/ --- (Updated June 5, 2014, 10:26 p.m.) Review request for giraph. Repository: giraph-git Description --- When some of the request processing threads fails, the worker gets stuck but the job doesn't fail and it has to be killed manually. We should detect netty thread crashes and fail the job automatically. Diffs - findbugs-exclude.xml e0466f7 giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyClient.java ae40c3b giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyMasterClient.java c982209 giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyMasterServer.java cb36c3e giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyServer.java 14d4ea8 giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java 7541418 giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java adb96cb giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/ExceptionHandler.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestServerHandler.java 601cd2f giraph-core/src/main/java/org/apache/giraph/graph/GraphMapper.java c86a024 giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java ad5fc91 giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 90dc9f3 giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java aff7084 giraph-core/src/main/java/org/apache/giraph/yarn/GiraphYarnTask.java f4719cc giraph-core/src/test/java/org/apache/giraph/comm/ConnectionTest.java e771e36 giraph-core/src/test/java/org/apache/giraph/comm/MockExceptionHandler.java PRE-CREATION giraph-core/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 236bc88 giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java fcdfa5c giraph-core/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java c026cf8 Diff: https://reviews.apache.org/r/21987/diff/ Testing --- Run some production jobs with this change. Also introduced random bugs in deserialization logic and confirmed that job fails. Thanks, Sergey Edunov
Re: Review Request 21987: Detect crashes of Netty threads
On May 29, 2014, 6:40 p.m., Pavan Kumar Athivarapu wrote: giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/ExceptionHandler.java, line 29 https://reviews.apache.org/r/21987/diff/1/?file=597792#file597792line29 this catches exceptions only for inbound conenctions, do u want to do this for outbound as well. in that case would it not be better to just piggy back on the counters override exceptioncaught in those methods? I am fine with creating new classes or just overriding the method in counters Pavan Kumar Athivarapu wrote: one more thing (for exception handling) - inbound handler should be at the end and outbound handler should be at the beginning of the pipeline so u catch exceptions at the end of processing input - 1 u catch exceptions at the end of sending output - 2 Sergey Edunov wrote: I felt like it's nice to have a separate class for exception handling as it outlines it's intention and makes it clear how and when to use it. even the new diff does not handle exceptions for outbound also please name is the handler to something like serverexceptionHandler, etc. instead of just exception - Pavan Kumar --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21987/#review44270 --- On June 5, 2014, 10:26 p.m., Sergey Edunov wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21987/ --- (Updated June 5, 2014, 10:26 p.m.) Review request for giraph. Repository: giraph-git Description --- When some of the request processing threads fails, the worker gets stuck but the job doesn't fail and it has to be killed manually. We should detect netty thread crashes and fail the job automatically. Diffs - findbugs-exclude.xml e0466f7 giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyClient.java ae40c3b giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyMasterClient.java c982209 giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyMasterServer.java cb36c3e giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyServer.java 14d4ea8 giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java 7541418 giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java adb96cb giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/ExceptionHandler.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestServerHandler.java 601cd2f giraph-core/src/main/java/org/apache/giraph/graph/GraphMapper.java c86a024 giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java ad5fc91 giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 90dc9f3 giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java aff7084 giraph-core/src/main/java/org/apache/giraph/yarn/GiraphYarnTask.java f4719cc giraph-core/src/test/java/org/apache/giraph/comm/ConnectionTest.java e771e36 giraph-core/src/test/java/org/apache/giraph/comm/MockExceptionHandler.java PRE-CREATION giraph-core/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 236bc88 giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java fcdfa5c giraph-core/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java c026cf8 Diff: https://reviews.apache.org/r/21987/diff/ Testing --- Run some production jobs with this change. Also introduced random bugs in deserialization logic and confirmed that job fails. Thanks, Sergey Edunov
[jira] [Commented] (GIRAPH-842) option to dump histogram of memory usage when heap is low on memory
[ https://issues.apache.org/jira/browse/GIRAPH-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025989#comment-14025989 ] Sergey Edunov commented on GIRAPH-842: -- Hey Pavan, this is great feature, here are some comments on implementation: 1) Please assign your thread some meaningful name, it can be helpful in debugging stack traces. 2) Stop flag has to be volatile, otherwise your thread may never see the change 3) I would let the thread die if you get InterruptedException. 4) runtime.freeMemory() might give you false alarms if you run job with different values of Xmx and Xms. This is because freeMemory only counts free bytes currently allocated to JVM. If your Xmx setting is bigger than Xms you can go low on freeMemory before JVM allocates another chunk. To get more accurate results you can do (maxMemory - totalMemory + freeMemory) option to dump histogram of memory usage when heap is low on memory --- Key: GIRAPH-842 URL: https://issues.apache.org/jira/browse/GIRAPH-842 Project: Giraph Issue Type: Bug Reporter: Pavan Kumar Assignee: Pavan Kumar Priority: Minor Attachments: GIRAPH-842.patch, master-stderr, worker-stderr Currently we are left in blind for jobs that OOM, it would be helpful if we can do a jmap -histo dump when heap has very little free space left. -- This message was sent by Atlassian JIRA (v6.2#6252)
