[
https://issues.apache.org/jira/browse/GIRAPH-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574230#comment-13574230
]
Hudson commented on GIRAPH-508:
-------------------------------
Integrated in Giraph-trunk-Commit #684 (See
[https://builds.apache.org/job/Giraph-trunk-Commit/684/])
GIRAPH-508: Increase the limit on the number of partitions (majakabiljo)
(Revision 8c86fa656d1dafeaba3fe55ac184ff3cbaa3e324)
Result = SUCCESS
majakabiljo :
http://git-wip-us.apache.org/repos/asf?p=giraph.git&a=commit&h=8c86fa656d1dafeaba3fe55ac184ff3cbaa3e324
Files :
* giraph-core/src/main/java/org/apache/giraph/partition/BasicPartitionOwner.java
*
giraph-core/src/main/java/org/apache/giraph/graph/AddressesAndPartitionsWritable.java
* CHANGELOG
* giraph-core/src/main/java/org/apache/giraph/partition/RangePartitionOwner.java
*
giraph-core/src/main/java/org/apache/giraph/partition/HashMasterPartitioner.java
* giraph-core/src/main/java/org/apache/giraph/partition/PartitionOwner.java
* giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java
> Increase the limit on the number of partitions
> ----------------------------------------------
>
> Key: GIRAPH-508
> URL: https://issues.apache.org/jira/browse/GIRAPH-508
> Project: Giraph
> Issue Type: Improvement
> Reporter: Maja Kabiljo
> Assignee: Maja Kabiljo
> Attachments: GIRAPH-508.diff
>
>
> We have the limit for total number of partitions of 2995. This is because of
> Zookeeper znode limit of 1MB, and from the assumption that partition owner
> description can take 300 bytes.
> In the simplest case, when checkpointing is not used and partitions don't
> move around, we have 5 ints and hostname written per partition. If partitions
> move around we have one more hostname and 2 ints. And when checkpointing is
> used we also have the path to checkpoint file written.
> For now, we can get rid of whole WorkerInfo description per partition, and
> just use taskIds, since all WorkerInfos are written in the beginning. This
> will lead to having just 4 ints per partition in the case when checkpointing
> is not used, and allow us to have much more partitions.
> When checkpointing is used, we can keep the limit (still up it a bit), or
> have all workers read partition metadata when restarting from checkpoint.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
