1b3MUw&r=70jRDqS1TgwYK0kwXqlG2wyrzJDvH1bm4B3mynUQPGE&m=be_cHeTDFl6onllF5gqZZLeEIXEyouhEQsSxvSLgle8&s=cK7a4D_MjmPorMJfXSbSW9Gw1914KdY4QMu0U3YkzZM&e=
>>>>>
>>>>>
>>>>> This release contains multiple new features. Among the most important
>&
+1
On 1/11/19, 10:13 AM, "Dionysios Logothetis" wrote:
+1 on migrating.
On Thu, Jan 3, 2019 at 5:19 AM Apache Infrastructure Team <
infrastruct...@apache.org> wrote:
> Hello, giraph folks.
> As stated earlier in 2018, all git repositories must be migrated from
http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java
1399356
Diff: https://reviews.apache.org/r/7610/diff/
Testing
---
Passes mvn verify
Thanks,
Maja Kabiljo
here.
- Maja
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7610/#review12540
---
On Oct. 17, 2012, 9:52 p.m., Maja
ous amount of
aggregators and wokrers.
Tested on fb application which uses a lot of big aggregators, also tested it
with multithreading.
Thanks,
Maja Kabiljo
application which uses a lot of big aggregators, also tested it
with multithreading.
Thanks,
Maja Kabiljo
/browse/GIRAPH-388?focusedCommentId=13484591
Thanks,
Maja Kabiljo
pulled out counters to separate classes. But why do you use
static instances of these classes, can't you have them as fields of
BspServiceMaster/MasterThread?
Also, it would be nicer if postApplication wasn't called from inside cleanup.
- Maja Kabiljo
On Oct. 29, 2012, 5:20 p.m., Nitay J
> On Nov. 2, 2012, 4:36 a.m., Alessandro Presta wrote:
> > I'm not sure how I feel about using MasterCompute for something other than
> > computation. I guess it gets the job done but I wonder if we should have a
> > better place for this sort of code (like a MasterContext or whatever). It's
>
https://reviews.apache.org/r/7883/diff/
Testing
---
mvn clean verify, pseudo-distributed tests
PageRankBenchmark (results above)
Tried it out with a real application
Thanks,
Maja Kabiljo
h/comm/RequestTest.java
1405925
Diff: https://reviews.apache.org/r/7883/diff/
Testing
---
mvn clean verify, pseudo-distributed tests
PageRankBenchmark (results above)
Tried it out with a real application
Thanks,
Maja Kabiljo
rg/r/7883/#review13115
---
On Nov. 5, 2012, 10:59 p.m., Maja Kabiljo wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://revi
li Reisman mailto:apache.mail...@gmail.com>>
Date: Tuesday, November 6, 2012 10:44 AM
To: "dev@giraph.apache.org<mailto:dev@giraph.apache.org>"
mailto:dev@giraph.apache.org>>, Maja Kabiljo
mailto:majakabi...@fb.com>>
Subject: Re: Review Request: More SendMessageCache
https://reviews.apache.org/r/7911/diff/
Testing (updated)
---
mvn clean verify
Run benchmarks on cluster with both values for giraph.splitMasterWorker
Thanks,
Maja Kabiljo
rstep time 55s->45s
6 compute threads, superstep time 28s->15s
12 compute threads, 1 netty server thread, superstep time 185s->112s
Our internal application
Similar speedup as Page Rank
Thanks,
Maja Kabiljo
akes sense to have some empty Combiner interface, and then
BinaryCombiner and MultiCombiner both extend it, and we can use that interface
on these problematic places?
I am also going to change all current combiners to BinaryCombiners.
- Maja Kabiljo
On Nov. 8, 2012, 10:56 p.m., Maja Kab
28s->15s
12 compute threads, 1 netty server thread, superstep time 185s->112s
Our internal application
Similar speedup as Page Rank
Thanks,
Maja Kabiljo
ostSuperstep, like in WorkerContext? A
lso, I think MasterObserver should at least be Configurable, or have a
GraphState in there, so we could access application state from implementing
classes.
- Maja Kabiljo
On Nov. 9, 2012, 12:29 a.m., Nitay
iews.apache.org/r/7975/diff/1/?file=187393#file187393line92>
> >
> > Shouldn't you change the "vertex combiner" terminology to "multi
> > combiner" across the board?
As I said in my comment, I haven't cleaned everything up because I am not s
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7981/#review13283
---
Ship it!
Thanks Nitay, +1.
- Maja Kabiljo
On Nov. 9, 2012, 3:09
java
<https://reviews.apache.org/r/7980/#comment28560>
You can use setValue here
giraph/src/main/java/org/apache/giraph/graph/Vertex.java
<https://reviews.apache.org/r/7980/#comment28559>
Nice refactoring.
- Maja Kabiljo
On Nov. 9, 2012, 12:07
etty server thread, superstep time 185s->112s
Our internal application
Similar speedup as Page Rank
Thanks,
Maja Kabiljo
intuitive interface.
> >
> > I'm also suggesting that the vertex index is not needed. I see the
> > parallel with the key in a MapReduce Reducer, but I'm not sure how it would
> > help here to know just the vertex id but not the value.
>
>
uperstep time 28s->15s
12 compute threads, 1 netty server thread, superstep time 185s->112s
Our internal application
Similar speedup as Page Rank
Thanks,
Maja Kabiljo
> On Nov. 9, 2012, 7:03 p.m., Maja Kabiljo wrote:
> > giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java, line 769
> > <https://reviews.apache.org/r/7980/diff/1/?file=187462#file187462line769>
> >
> > You can use setValue here
>
>
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7980/#review13368
---
Ship it!
+1, thanks Nitay.
- Maja Kabiljo
On Nov. 12, 2012, 6:59
e 28s->15s
12 compute threads, 1 netty server thread, superstep time 185s->112s
Our internal application
Similar speedup as Page Rank
Thanks,
Maja Kabiljo
sting
---
mvn clean verify
Run benchmarks on cluster with both values for giraph.splitMasterWorker
Thanks,
Maja Kabiljo
org/apache/giraph/utils/WritableUtils.java
<https://reviews.apache.org/r/7990/#comment28632>
Why do we always write -1 here and ignore the first int value when reading?
- Maja Kabiljo
On Nov. 14, 2012, 12:43 a.m., Avery Ching wrote:
>
> ---
improvements on this field!
- Maja Kabiljo
On Nov. 14, 2012, 3:06 a.m., Avery Ching wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apach
.apache.org/r/8042/#comment29021>
Revisit the comments
- Maja Kabiljo
On Nov. 16, 2012, 8:18 p.m., Nitay Joffe wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8042/#review13602
---
Looks good now.
- Maja Kabiljo
On Nov. 20, 2012, 12:08 a.m
omm/messages/SequentialFileMessageStore.java
<https://reviews.apache.org/r/8140/#comment29258>
Write the number of messages?
http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/ComputeCallable.java
<https://reviews.apache.org/r/8140/#comm
Can we
do better?
http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/CollectionUtils.java
<https://reviews.apache.org/r/8140/#comment29319>
Probably not important, but this won't work correctly if we have duplicates
in the iterable. Maybe
repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java
<https://reviews.apache.org/r/8140/#comment29453>
Change to ByteArrayMessagesPerVertexStore
- Maja Kabiljo
On Nov. 27, 2012, 7:52 a.m., Avery
giraph/src/test/java/org/apache/giraph/comm/RequestTest.java
1415035
http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java
1415035
Diff: https://reviews.apache.org/r/8286/diff/
Testing
---
mvn verify
Thanks,
Maja Kabiljo
It's also nicer this way for WaitableWithoutResult.
- Maja
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8286/#review13886
---------
pache.org/r/8286/#review13885
---
On Nov. 29, 2012, 11:59 p.m., Maja Kabiljo wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://revi
sting
---
Using this patch, here is the performance gain on one of our large applications:
- about 10% of overall superstep time
- about 25% of compute time
- about 10% less GC
Passes mvn verify.
Thanks,
Maja Kabiljo
view14061
---
On Dec. 5, 2012, 8:05 p.m., Maja Kabiljo wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://revi
/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java
1417645
Diff: https://reviews.apache.org/r/8407/diff/
Testing
---
mvn verify
Large application run on a cluster
Thanks,
Maja Kabiljo
omment.
http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java
<https://reviews.apache.org/r/8417/#comment30297>
Maybe rename this variable - it's the moment in which we want to give up,
not the number of msecs left.
- Maja
n run on a cluster
Thanks,
Maja Kabiljo
it:
https://reviews.apache.org/r/8407/#review14208
---
On Dec. 10, 2012, 6:07 p.m., Maja Kabiljo wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://revi
patch - about 310s for input split threads to finish, 500s for
input superstep
Thanks,
Maja Kabiljo
this patch - about 310s for input split threads to finish, 500s for
input superstep
Thanks,
Maja Kabiljo
/giraph/partition/PartitionOwner.java
a886d79
giraph-core/src/main/java/org/apache/giraph/partition/RangePartitionOwner.java
1ecedb8
Diff: https://reviews.apache.org/r/9370/diff/
Testing
---
Run on real application with 200 workers and 40k partitions
mvn clean verify
Thanks,
Maja
/test/java/org/apache/giraph/TestPartitionContext.java
PRE-CREATION
Diff: https://reviews.apache.org/r/9372/diff/
Testing
---
Added test for PartitionContext
mvn clean verify
Thanks,
Maja Kabiljo
/giraph/BspCase.java 6aab533
giraph-core/src/test/java/org/apache/giraph/TestPartitionContext.java
PRE-CREATION
Diff: https://reviews.apache.org/r/9372/diff/
Testing
---
Added test for PartitionContext
mvn clean verify
Thanks,
Maja Kabiljo
tps://reviews.apache.org/r/9372/#review16347
-------
On Feb. 8, 2013, 5:30 p.m., Maja Kabiljo wrote:
>
> ---
> This is an automatically generated e-mail.
tions.
- Maja
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9372/#review16349
---
On Feb. 8, 2013, 5:30 p.m., Maj
copy of
SendMessageCache, replacing M with E and ByteArrayVertexIdMessages with
ByteArrayVertexIdEdges (which are also very similar).
- Maja Kabiljo
On Feb. 14, 2013, 6:34 p.m., Alessandro Presta wrote:
>
> ---
> This is an
/org/apache/giraph/comm/requests/SendWorkerMessagesRequest.java
<https://reviews.apache.org/r/9449/#comment35348>
In order to avoid casts like this, you can make SendWorkerDataRequest>
- Maja Kabiljo
On Feb. 15, 2013, 6:17 p.m., Alessandro Pre
Also you can try decreasing giraph.maxRequestMilliseconds and see what
happens when requests get resent. If you have a problem on the worker they
won't arrive again, if it is some concurrency issue then they should
arrive on second try.
On 2/19/13 3:18 PM, "Avery Ching" wrote:
>Try these to star
Sebastian, how many input splits do you have? What is the last line you
see in the logs? We had some problems with zookeeper whit a lot of input
splits (order of 10k), which we fixed by increasing the heap size for
zookeeper (changing giraph.zkJavaOpts). You can try doing that and see if
it helps.
Yeah, that's exactly what I told you.
We have one ExtendedDataOutput per vertex. And the limit is about
(2^30)/4, since you will have double of the size allocated at some point.
What do you mean you don't want to use a combiner for evaluation reasons?
On 2/21/13 5:38 AM, "Claudio Martella" wrote:
giraph/hive/input/edge/HiveEdgeReader.java
<https://reviews.apache.org/r/8611/#comment35800>
Could we have an option to reuse edge objects here?
- Maja Kabiljo
On Feb. 21, 2013, 6:17 p.m., Nitay Joffe wrote:
>
> ---
>
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8611/#review16868
---
Ship it!
Forgot to say, I'm +1 on this :-)
- Maja Kabiljo
O
a/org/apache/giraph/worker/WorkerAggregatorHandler.java
001cf59
Diff: https://reviews.apache.org/r/10029/diff/
Testing
---
mvn clean install
AggregatorsBenchmark
Thanks,
Maja Kabiljo
then we are not going to receive any aggregator request from them,
and will see this kind of log.
- Maja
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10029/#review18105
---
On March 19, 201
.java
PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/worker/WorkerAggregatorHandler.java
001cf59
Diff: https://reviews.apache.org/r/10029/diff/
Testing
---
mvn clean install
AggregatorsBenchmark
Thanks,
Maja Kabiljo
waiting on, when they fall under a certain
> > number (like we do with other requests), since we can't see that in the
> > master status anymore.
>
> Maja Kabiljo wrote:
> I understood that, it's what this patch is providing :-)
> Here is the
.java
PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/worker/WorkerAggregatorHandler.java
001cf59
Diff: https://reviews.apache.org/r/10029/diff/
Testing
---
mvn clean install
AggregatorsBenchmark
Thanks,
Maja Kabiljo
waiting on, when they fall under a certain
> > number (like we do with other requests), since we can't see that in the
> > master status anymore.
>
> Maja Kabiljo wrote:
> I understood that, it's what this patch is providing :-)
> Here is the
https://reviews.apache.org/r/10126/diff/
Testing
---
Thanks,
Maja Kabiljo
org/apache/giraph/hive/output/AbstractVertexToHive.java
8e3f1ca
Diff: https://reviews.apache.org/r/10126/diff/
Testing
---
Thanks,
Maja Kabiljo
impleNoEdgesHiveToVertex is for the cases when we have no edges.
- Maja
-------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10126/#review18414
---
dges in edge input, so you can clean up this part
(though no need if GIRAPH-588 is going to get in :-))
- Maja Kabiljo
On March 27, 2013, 7:14 a.m., Nitay Joffe wrote:
>
> ---
> This is an automatically generated e-mail. To
always reuse edge objects in edge input. But
for vertex input (currently HiveToVertexEdges) we want to have this
infromation, right?
- Maja Kabiljo
On March 27, 2013, 7:12 a.m., Nitay Joffe wrote:
>
> ---
> This is an automatically
reuse, but
with vertex input currently we can't, right? So SimpleHiveToVertex needs this
option. I can fix it, or you can put it back as part of this patch, let me know
either way.
- Maja Kabiljo
On March 28, 2013, 4:03 a.m., Nitay Joffe
---
mvn clean install
Real application using SynchronizedSuperstepOutput
Thanks,
Maja Kabiljo
-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java
74c1f87
Diff: https://reviews.apache.org/r/10201/diff/
Testing
---
mvn clean install
Real application using SynchronizedSuperstepOutput
Thanks,
Maja Kabiljo
eview!
- Maja
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10201/#review18537
---
On March 30, 2013, 2:11 a.m., Maj
option to dump
these output files per superstep.
- Maja
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10201/#review18554
-------
asf/giraph/tree/da2a6870
>> Diff: http://git-wip-us.apache.org/repos/asf/giraph/diff/da2a6870
>>
>> Branch: refs/heads/trunk
>> Commit: da2a68708b69d984a3f91710cc6445845cc086b2
>> Parents: 0e04147
>> Author: M
iraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/SimpleNoEdgesHiveToVertex.java
ca427a0
Diff: https://reviews.apache.org/r/10248/diff/
Testing
---
mvn clean install
Thanks,
Maja Kabiljo
.apache.org/r/10248/#review18631
---
On April 3, 2013, 3:51 a.m., Maja Kabiljo wrote:
>
> ---
> This is an automatically generated e-mail. To reply, vi
.java
ca427a0
Diff: https://reviews.apache.org/r/10248/diff/
Testing
---
mvn clean install
Thanks,
Maja Kabiljo
/RandomMessageBenchmark.java
ff65ab9
giraph-core/src/main/java/org/apache/giraph/benchmark/ShortestPathsBenchmark.java
c3c714e
Diff: https://reviews.apache.org/r/10285/diff/
Testing
---
mvn clean install, run a few benchmarks
Thanks,
Maja Kabiljo
ormat.java
0117ce9
Diff: https://reviews.apache.org/r/10343/diff/
Testing
---
Thanks,
Maja Kabiljo
ank runs on Giraph.
I'll put in the locality.
- Maja
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10343/#review18791
---------
a
0117ce9
Diff: https://reviews.apache.org/r/10343/diff/
Testing
---
Thanks,
Maja Kabiljo
/r/10389/diff/
Testing
---
mvn clean install
Tested on real application with all options (set number of output threads with
an without thread safe output), using giraph-hive.
Thanks,
Maja Kabiljo
n automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10389/#review18960
---
On April 10, 2013, 12:47 a.m., Maja Kabiljo wrote:
>
> ---
> This is an a
ph/io/formats/PseudoRandomVertexInputFormat.java
dca0271
giraph-core/src/test/java/org/apache/giraph/io/TestJsonBase64Format.java
0117ce9
Diff: https://reviews.apache.org/r/10343/diff/
Testing
---
Thanks,
Maja Kabiljo
There is still the output in WeightedPageRankBenchmark, I guess for the
case Alessandro mentioned. Tried to keep PageRankBenchmark as simple as
possible.
On 4/11/13 4:12 PM, "Alessandro Presta" wrote:
>As far as I know, benchmarks never supported output. What's the use case?
>Comparing the outpu
476cd
Diff: https://reviews.apache.org/r/10564/diff/
Testing
---
mvn clean verify
Run application with two edge input tables - verified results.
Thanks,
Maja Kabiljo
a
c482cf0
giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/HiveEdgeReader.java
09476cd
Diff: https://reviews.apache.org/r/10564/diff/
Testing
---
mvn clean verify
Run application with two edge input tables - verified results.
Thanks,
Maja Kabiljo
ally generated e-mail. To reply, visit:
https://reviews.apache.org/r/10564/#review19285
---
On April 17, 2013, 4:19 a.m., Maja Kabiljo wrote:
>
> ---
> This is an autom
.
- Maja
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10564/#review19322
---
On April 17, 2013, 4:19 a.m., Maja Kabiljo wrote:
>
/apache/giraph/edge/LongNullHashSetEdges.java
ef0302c
Diff: https://reviews.apache.org/r/10598/diff/
Testing
---
mvn clean verify
Application run which requires this kind of initialization
Thanks,
Maja Kabiljo
a/org/apache/giraph/edge/LongNullHashSetEdges.java
ef0302c
giraph-core/src/main/java/org/apache/giraph/utils/EdgeIterables.java e945d7b
Diff: https://reviews.apache.org/r/10598/diff/
Testing
---
mvn clean verify
Application run which requires this kind of initialization
Thanks,
e/src/main/java/org/apache/giraph/hive/output/HiveVertexOutputFormat.java
45c9ca3
giraph-hive/src/main/java/org/apache/giraph/hive/output/HiveVertexWriter.java
0215428
Diff: https://reviews.apache.org/r/10690/diff/
Testing
---
mvn clean verify
Real application run with hive io
Thanks,
/worker/EdgeInputSplitsCallable.java
de1e774
giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallableFactory.java
4a1705b
giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java
8d67c1d
Diff: https://reviews.apache.org/r/10564/diff/
Testing
---
mvn clean verify
Run application with two edge input tables - verified results.
Thanks,
Maja Kabiljo
iraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java
8d67c1d
Diff: https://reviews.apache.org/r/10564/diff/
Testing
---
mvn clean verify
Run application with two edge input tables - verified results.
Thanks,
Maja Kabiljo
/WrappedVertexReader.java
PRE-CREATION
Diff: https://reviews.apache.org/r/10958/diff/
Testing
---
Job run with 20 threads
Thanks,
Maja Kabiljo
tices' instead of 'vertexes'.
giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java
<https://reviews.apache.org/r/10937/#comment41465>
Add the leftover in inputSplitVerticesFiltered in the end?
- Maja Kabiljo
On May 3, 2013, 10:
er than'?
'a couple OF lines'?
- Maja Kabiljo
On May 6, 2013, 9:22 p.m., Avery Ching wrote:
>
> ---
> This is an automatically generated
960/#comment41543>
One more thing: the way I see it the first paragraph of the landing page
should be something simpler and not go into background details that much. What
do you think?
- Maja Kabiljo
On May 6, 2013, 10:04 p.m., Avery Ching
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11107/#review20502
---
Ship it!
Good find, +1!
- Maja Kabiljo
On May 13, 2013, 8:24
1 - 100 of 992 matches
Mail list logo
