Programmer can't control java memory like malloc/ free in c, type boxing/ unboxing, etc., it seems not be easy to evaluate the memory. So it would be good sticking to erlang fail fast style. Or we can have a programme that load data and measure the actual memory usage.
On 24 February 2014 22:32, Tommaso Teofili <tommaso.teof...@gmail.com> wrote: > 2014-02-24 13:52 GMT+01:00 Edward J. Yoon <edwardy...@apache.org>: > >> 0.6.4 or 0.7.0, Both are OK to me. >> >> Just FYI, >> >> The memory efficiency has been significantly (almost x2-3) improved by >> runtime message serialization and compression. See >> >> https://wiki.apache.org/hama/Benchmarks#PageRank_Performance_0.7.0-SNAPSHOT_vs_0.6.3 >> (I'll attach more benchmarks and comparisons with other systems result >> soon). And, we've fixed many bugs. e.g., K-Means, NeuralNetwork, >> SemiClustering, Graph's Combiners HAMA-857. >> > > sure, all the above things look good to me. > > >> >> According to my personal evaluations, current system is fairly >> respectable. As I mentioned before, I believe we should stick to >> in-memory style since the today's machines can be equipped with up to >> 128 GB. Disk (or disk hybrid) based queue is a optional, not a >> must-have. >> > > right, the only thing that I think we need to address before 0.7.0 is > related to the OutOfMemory errors (especially when dealing with large > graphs); for example IMHO even if the memory is not enough to store all the > graph vertices assigned to a certain peer, a scalable system should never > throw OOM exceptions, instead it may eventually process items slower (with > caches / queues) but never throw an exception for that but that's just my > opinion. > > >> >> Once we release this one, we finally might want to focus on below issues: >> >> * Fault tolerant job processing (checkpoint recovery) >> > > +1 > > >> * Support GPUs and InfiniBand >> > > +1 for the former, not sure about the latter. > > >> >> Then, I think we can release version 1.0. >> > > My 2 cents, > Tommaso > > >> >> On Mon, Feb 24, 2014 at 8:44 PM, Tommaso Teofili >> <tommaso.teof...@gmail.com> wrote: >> > Would you cut 0.7 or 0.6.4 ? >> > I'd go with 0.6.4 as I think the next minor version change should be due >> to >> > significant feature additions / changes and / or stability / scalability >> > improvements. >> > >> > Regards, >> > Tommaso >> > >> > >> > 2014-02-24 8:47 GMT+01:00 Edward J. Yoon <edwardy...@apache.org>: >> > >> >> Hi all, >> >> >> >> I plan on cutting a release next week. If you have some opinions, Pls >> feel >> >> free to comment here. >> >> >> >> Sent from my iPhone >> >> >> >> -- >> Edward J. Yoon (@eddieyoon) >> Chief Executive Officer >> DataSayer, Inc. >>
