[jira] [Created] (GIRAPH-915) With BigDataIO some messages can get ignored
Maja Kabiljo created GIRAPH-915: --- Summary: With BigDataIO some messages can get ignored Key: GIRAPH-915 URL: https://issues.apache.org/jira/browse/GIRAPH-915 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Assignee: Maja Kabiljo ExtendedDataInput has int available() method, and when iterating through messages we check if it's grater then zero. For BigDataIO we can get an overflow here, returning a negative values and ignoring the rest of messages. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (GIRAPH-915) With BigDataIO some messages can get ignored
[ https://issues.apache.org/jira/browse/GIRAPH-915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-915: Attachment: GIRAPH-915.patch Replacing available() method with method which checks whether we are at the end of input, since that's the only way in which we use it anyways. Tested on a job which had missing messages before and verified that it works now. With BigDataIO some messages can get ignored Key: GIRAPH-915 URL: https://issues.apache.org/jira/browse/GIRAPH-915 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Assignee: Maja Kabiljo Attachments: GIRAPH-915.patch ExtendedDataInput has int available() method, and when iterating through messages we check if it's grater then zero. For BigDataIO we can get an overflow here, returning a negative values and ignoring the rest of messages. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (GIRAPH-916) Wrong number of vertices stored reported to command line
Maja Kabiljo created GIRAPH-916: --- Summary: Wrong number of vertices stored reported to command line Key: GIRAPH-916 URL: https://issues.apache.org/jira/browse/GIRAPH-916 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Assignee: Maja Kabiljo Priority: Minor Attachments: GIRAPH-916.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (GIRAPH-916) Wrong number of vertices stored reported to command line
[ https://issues.apache.org/jira/browse/GIRAPH-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-916: Attachment: GIRAPH-916.patch Wrong number of vertices stored reported to command line Key: GIRAPH-916 URL: https://issues.apache.org/jira/browse/GIRAPH-916 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Assignee: Maja Kabiljo Priority: Minor Attachments: GIRAPH-916.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GIRAPH-915) With BigDataIO some messages can get ignored
[ https://issues.apache.org/jira/browse/GIRAPH-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026786#comment-14026786 ] Pavan Kumar commented on GIRAPH-915: can't u just leave available as is, and define the other another method. otherwise lgtm With BigDataIO some messages can get ignored Key: GIRAPH-915 URL: https://issues.apache.org/jira/browse/GIRAPH-915 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Assignee: Maja Kabiljo Attachments: GIRAPH-915.patch ExtendedDataInput has int available() method, and when iterating through messages we check if it's grater then zero. For BigDataIO we can get an overflow here, returning a negative values and ignoring the rest of messages. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GIRAPH-916) Wrong number of vertices stored reported to command line
[ https://issues.apache.org/jira/browse/GIRAPH-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026793#comment-14026793 ] Pavan Kumar commented on GIRAPH-916: +1. lgtm Wrong number of vertices stored reported to command line Key: GIRAPH-916 URL: https://issues.apache.org/jira/browse/GIRAPH-916 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Assignee: Maja Kabiljo Priority: Minor Attachments: GIRAPH-916.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (GIRAPH-842) option to dump histogram of memory usage when heap is low on memory
[ https://issues.apache.org/jira/browse/GIRAPH-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavan Kumar updated GIRAPH-842: --- Attachment: GIRAPH-842_1.patch addressed comments. pl take a look. Thanks option to dump histogram of memory usage when heap is low on memory --- Key: GIRAPH-842 URL: https://issues.apache.org/jira/browse/GIRAPH-842 Project: Giraph Issue Type: Bug Reporter: Pavan Kumar Assignee: Pavan Kumar Priority: Minor Attachments: GIRAPH-842.patch, GIRAPH-842_1.patch, master-stderr, worker-stderr Currently we are left in blind for jobs that OOM, it would be helpful if we can do a jmap -histo dump when heap has very little free space left. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GIRAPH-915) With BigDataIO some messages can get ignored
[ https://issues.apache.org/jira/browse/GIRAPH-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026846#comment-14026846 ] Maja Kabiljo commented on GIRAPH-915: - But available can't be implemented correctly in BigDataIO. I could throw an exception there if it sounds better. And available() is not going to be used anywhere. What do you think? With BigDataIO some messages can get ignored Key: GIRAPH-915 URL: https://issues.apache.org/jira/browse/GIRAPH-915 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Assignee: Maja Kabiljo Attachments: GIRAPH-915.patch ExtendedDataInput has int available() method, and when iterating through messages we check if it's grater then zero. For BigDataIO we can get an overflow here, returning a negative values and ignoring the rest of messages. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GIRAPH-842) option to dump histogram of memory usage when heap is low on memory
[ https://issues.apache.org/jira/browse/GIRAPH-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026848#comment-14026848 ] Sergey Edunov commented on GIRAPH-842: -- +1 option to dump histogram of memory usage when heap is low on memory --- Key: GIRAPH-842 URL: https://issues.apache.org/jira/browse/GIRAPH-842 Project: Giraph Issue Type: Bug Reporter: Pavan Kumar Assignee: Pavan Kumar Priority: Minor Attachments: GIRAPH-842.patch, GIRAPH-842_1.patch, master-stderr, worker-stderr Currently we are left in blind for jobs that OOM, it would be helpful if we can do a jmap -histo dump when heap has very little free space left. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GIRAPH-915) With BigDataIO some messages can get ignored
[ https://issues.apache.org/jira/browse/GIRAPH-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026857#comment-14026857 ] Pavan Kumar commented on GIRAPH-915: throwing an exception sounds better. this is because we can have future implementations of bytedatastructs and available is really needed (for example in bytebufdatainput, etc.) With BigDataIO some messages can get ignored Key: GIRAPH-915 URL: https://issues.apache.org/jira/browse/GIRAPH-915 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Assignee: Maja Kabiljo Attachments: GIRAPH-915.patch ExtendedDataInput has int available() method, and when iterating through messages we check if it's grater then zero. For BigDataIO we can get an overflow here, returning a negative values and ignoring the rest of messages. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GIRAPH-842) option to dump histogram of memory usage when heap is low on memory
[ https://issues.apache.org/jira/browse/GIRAPH-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14027014#comment-14027014 ] Hudson commented on GIRAPH-842: --- ABORTED: Integrated in Giraph-trunk-Commit #1451 (See [https://builds.apache.org/job/Giraph-trunk-Commit/1451/]) GIRAPH-842: option to dump histogram of memory usage when heap is low on memory (pavanka) (pavanka: http://git-wip-us.apache.org/repos/asf?p=giraph.gita=commith=372c35ebccedf64266a38a35e365d3810642fef2) * CHANGELOG * giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java * giraph-core/src/main/java/org/apache/giraph/utils/ReactiveJMapHistoDumper.java * giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java * giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java * giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java option to dump histogram of memory usage when heap is low on memory --- Key: GIRAPH-842 URL: https://issues.apache.org/jira/browse/GIRAPH-842 Project: Giraph Issue Type: Bug Reporter: Pavan Kumar Assignee: Pavan Kumar Priority: Minor Attachments: GIRAPH-842.patch, GIRAPH-842_1.patch, master-stderr, worker-stderr Currently we are left in blind for jobs that OOM, it would be helpful if we can do a jmap -histo dump when heap has very little free space left. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (GIRAPH-917) Every commit is followed by ABORTED: Integrated in Giraph-trunk-Commit
Pavan Kumar created GIRAPH-917: -- Summary: Every commit is followed by ABORTED: Integrated in Giraph-trunk-Commit Key: GIRAPH-917 URL: https://issues.apache.org/jira/browse/GIRAPH-917 Project: Giraph Issue Type: Task Reporter: Pavan Kumar Currently, every continuous build is aborted, because it is taking too long. Shouldn't we reduce the testing time so we can be with in the time limit or have a way to increase the time limits. Would be great if some one from PMC can take a look. This issue has been bugging us for very long now. Continuous build has become useless in verifying correctness of commit because of this issue. Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (GIRAPH-918) GIRAPH-908 has a small bug reg counting entries
Pavan Kumar created GIRAPH-918: -- Summary: GIRAPH-908 has a small bug reg counting entries Key: GIRAPH-918 URL: https://issues.apache.org/jira/browse/GIRAPH-918 Project: Giraph Issue Type: Bug Reporter: Pavan Kumar Assignee: Pavan Kumar Priority: Trivial Fix trivial counting bug (int used instead of long) Also remove mappingstore earlier (before moving edges to vertices) to be gc friendly -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (GIRAPH-918) GIRAPH-908 has a small bug reg counting entries
[ https://issues.apache.org/jira/browse/GIRAPH-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavan Kumar updated GIRAPH-918: --- Attachment: GIRAPH-918.patch trivial change GIRAPH-908 has a small bug reg counting entries --- Key: GIRAPH-918 URL: https://issues.apache.org/jira/browse/GIRAPH-918 Project: Giraph Issue Type: Bug Reporter: Pavan Kumar Assignee: Pavan Kumar Priority: Trivial Attachments: GIRAPH-918.patch Fix trivial counting bug (int used instead of long) Also remove mappingstore earlier (before moving edges to vertices) to be gc friendly -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GIRAPH-918) GIRAPH-908 has a small bug reg counting entries
[ https://issues.apache.org/jira/browse/GIRAPH-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14027241#comment-14027241 ] Hudson commented on GIRAPH-918: --- ABORTED: Integrated in Giraph-trunk-Commit #1452 (See [https://builds.apache.org/job/Giraph-trunk-Commit/1452/]) GIRAPH-918: GIRAPH-908 has a small bug reg counting entries (pavanka) (pavanka: http://git-wip-us.apache.org/repos/asf?p=giraph.gita=commith=ffdddff32e7ac6e2cbb5ae8471c1192c69ac6a94) * CHANGELOG * giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java GIRAPH-908 has a small bug reg counting entries --- Key: GIRAPH-918 URL: https://issues.apache.org/jira/browse/GIRAPH-918 Project: Giraph Issue Type: Bug Reporter: Pavan Kumar Assignee: Pavan Kumar Priority: Trivial Attachments: GIRAPH-918.patch Fix trivial counting bug (int used instead of long) Also remove mappingstore earlier (before moving edges to vertices) to be gc friendly -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (GIRAPH-908) support for partitioned input in giraph
[ https://issues.apache.org/jira/browse/GIRAPH-908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14027242#comment-14027242 ] Hudson commented on GIRAPH-908: --- ABORTED: Integrated in Giraph-trunk-Commit #1452 (See [https://builds.apache.org/job/Giraph-trunk-Commit/1452/]) GIRAPH-918: GIRAPH-908 has a small bug reg counting entries (pavanka) (pavanka: http://git-wip-us.apache.org/repos/asf?p=giraph.gita=commith=ffdddff32e7ac6e2cbb5ae8471c1192c69ac6a94) * CHANGELOG * giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java support for partitioned input in giraph --- Key: GIRAPH-908 URL: https://issues.apache.org/jira/browse/GIRAPH-908 Project: Giraph Issue Type: Sub-task Reporter: Pavan Kumar Assignee: Pavan Kumar Attachments: GIRAPH-908.patch When the graph we need to work on is already partitioned into a fixed number of buckets, with properties such as high edge-locality, low fan-out to other buckets, etc. [for instance using techniques such as https://people.cam.cornell.edu/~jugander/papers/wsdm13-blp.pdf ] we should be able to partition our graph based on such a mapping, to improve local-requests, etc. thus avoiding huge network communication. this diff is especially useful we repeatedly run algorithms on top of the same graph. In such case, we can compute the partitioning once then use it to speed up processing, (also requiring less network bandwidth, etc.) for the rest applications on same/ similar graphs. The diff is big partly reviewed by my colleagues. Putting it up for review -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (GIRAPH-915) With BigDataIO some messages can get ignored
[ https://issues.apache.org/jira/browse/GIRAPH-915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-915: Attachment: GIRAPH-915.patch Returned available() method, rebased With BigDataIO some messages can get ignored Key: GIRAPH-915 URL: https://issues.apache.org/jira/browse/GIRAPH-915 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Assignee: Maja Kabiljo Attachments: GIRAPH-915.patch, GIRAPH-915.patch ExtendedDataInput has int available() method, and when iterating through messages we check if it's grater then zero. For BigDataIO we can get an overflow here, returning a negative values and ignoring the rest of messages. -- This message was sent by Atlassian JIRA (v6.2#6252)
