There is a bug in ArrayListWritable – readFileds should clear the list before reading any data to it. We are reusing message objects and every time we call readFields data is just going to be appended to the previous message. It's very easy to fix, Kyle please feel free to open an issue and submit a patch for it.
From: Kyle Orlando <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Tuesday, August 13, 2013 3:30 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Problems with sending/receiving messages of type IntArrayListWritable Tried it, got the same exact results. Do you think it's a problem with ArrayListWritable, or with sendMessage()? I don't have a great understanding of how sendMessage() utilizes the Writable interface exactly... I'm guessing that at some point it uses readFields() and write() because those obviously need to be implemented for a class to be considered "Writable". Looking at those two methods for ArrayListWritable, nothing jumps out at me as being a bug, though I can see where the minimum number of values in the ArrayList could be set incorrectly in readFields(): int numValues = in.readInt(); // read number of values ensureCapacity(numValues); What do you think? On Tue, Aug 13, 2013 at 5:50 PM, Claudio Martella <[email protected]<mailto:[email protected]>> wrote: could you try using: -D giraph.oneToAllMsgSending=true when you run it? On Tue, Aug 13, 2013 at 11:41 PM, Kyle Orlando <[email protected]<mailto:[email protected]>> wrote: Yep, and I've pulled recently as well. On Tue, Aug 13, 2013 at 3:10 PM, Claudio Martella <[email protected]<mailto:[email protected]>> wrote: That's strange. This looks like a bug to me. Are you using trunk? On Mon, Aug 12, 2013 at 4:32 PM, Kyle Orlando <[email protected]<mailto:[email protected]>> wrote: Also, I realize that I just sent an uncommented version... sorry guys, but the algorithm consists of just three supersteps: Superstep 0.) Source sends list of source's neighbors to each neighbor Superstep 1.) Neighbor receives list of source's neighbors from source, counts number of mutual neighbors, sends this value (in a 1-element list) back to source Superstep 2.) Source receives messages from neighbors that contain number of mutual neighbors. These are summed up and divided by the number of possible connections between neighbors to get the local clustering coefficient (between 0 and 1). This is set as the source vertex's value. On Mon, Aug 12, 2013 at 7:28 AM, Kyle Orlando <[email protected]<mailto:[email protected]>> wrote: > Alrighty, here's my code: > > public void compute( > Vertex<IntWritable, DoubleWritable, NullWritable> > vertex, > Iterable<IntArrayListWritable> messages) throws > IOException { > > if (isSource(vertex)) { > if (getSuperstep() == 0) { > System.out.println("\nSUPERSTEP > 0\n------------------------------"); > IntArrayListWritable sourceNeighbors = new > IntArrayListWritable(); > > vertex.setValue(new DoubleWritable(0)); > > for (Edge<IntWritable, NullWritable> edge : > vertex.getEdges()) { > IntWritable targetVertex = new > IntWritable(edge.getTargetVertexId().get()); > sourceNeighbors.add(targetVertex); > } > > for (IntWritable neighbor : > sourceNeighbors) { > sendMessage(neighbor, > sourceNeighbors); > System.out.println("Vertex " + > vertex.getId() + " sends " + > sourceNeighbors + " to " + neighbor); > } > > } else if (getSuperstep() == 2) { > int total = vertex.getNumEdges(); > int possibleCombos = total * (total - 1); > int mutualNeighbors = 0; > double coefficient = 0; > int count = 1; > > System.out.println("\nSUPERSTEP > 2\n------------------------------"); > > for (IntArrayListWritable message : > messages) { > System.out.println("Message " + > count + " contains: " + message); > mutualNeighbors += > message.get(0).get(); > count++; > } > > if (possibleCombos > 0) { > coefficient = > (double)mutualNeighbors / possibleCombos; > vertex.setValue(new > DoubleWritable(coefficient)); > } > > } > } else { > vertex.setValue(new DoubleWritable(0)); > > if (getSuperstep() == 1) { > int mutualNeighbors = 0; > > System.out.println("\nSUPERSTEP > 1\n------------------------------"); > IntArrayListWritable sourceNeighbors = > messages.iterator().next(); > System.out.println("Vertex " + > vertex.getId() + " receives " + > sourceNeighbors); > > for (Edge<IntWritable, NullWritable> edge : > vertex.getEdges()) { > IntWritable targetVertex = new > IntWritable(edge.getTargetVertexId().get()); > if > (sourceNeighbors.contains(targetVertex)) mutualNeighbors++; > } > > IntArrayListWritable neighborCount = new > IntArrayListWritable(); > neighborCount.add(new > IntWritable(mutualNeighbors)); > > sendMessage(new > IntWritable(SOURCE_ID.get(getConf())), neighborCount); > System.out.println("Vertex " + > vertex.getId() + " sends " + > neighborCount + " to Vertex " + SOURCE_ID.get(getConf())); > > } > } > > vertex.voteToHalt(); > } > > > On Fri, Aug 9, 2013 at 10:15 PM, Claudio Martella > <[email protected]<mailto:[email protected]>> wrote: >> you'd have to show us the code in the compute method. your problem might be >> caused by object reusal. >> >> >> On Fri, Aug 9, 2013 at 9:05 PM, Kyle Orlando >> <[email protected]<mailto:[email protected]>> >> wrote: >>> >>> Hello, >>> >>> I am trying to write code to compute the local clustering coefficient >>> of a vertex/some vertices, and to do this I send a message that >>> contains a list of the source's neighbors to each of its neighbors. >>> This is in, of course, an IntArrayListWritable. I check the list that >>> I am sending before invoking sendMessage(), and it appears to be >>> correct. However, when I use message.iterator.next() or something >>> similar, the IntArrayListWritable objects seem to repeat or something. >>> I did some logging, and here was the output: >>> >>> SUPERSTEP 0 >>> ------------------------------ >>> Vertex 1 sends [2, 3, 4] to Vertex 2 >>> Vertex 1 sends [2, 3, 4] to Vertex 3 >>> Vertex 1 sends [2, 3, 4] to Vertex 4 >>> >>> SUPERSTEP 1 >>> ------------------------------ >>> Vertex 2 receives [2, 3, 4] >>> Vertex 2 sends [1] to Vertex 1 >>> >>> SUPERSTEP 1 >>> ------------------------------ >>> Vertex 3 receives [2, 3, 4, 2, 3, 4] >>> Vertex 3 sends [1] to Vertex 1 >>> >>> SUPERSTEP 1 >>> ------------------------------ >>> Vertex 4 receives [2, 3, 4, 2, 3, 4, 2, 3, 4] >>> Vertex 4 sends [0] to Vertex 1 >>> >>> SUPERSTEP 2 >>> ------------------------------ >>> Message 1 contains: [1] >>> Message 2 contains: [1, 1, 1] >>> Message 3 contains: [1, 1, 1, 1, 1, 0] >>> >>> What is happening? >>> >>> -- >>> Kyle Orlando >>> Computer Engineering Major >>> University of Maryland >> >> >> >> >> -- >> Claudio Martella >> [email protected]<mailto:[email protected]> > > > > -- > Kyle Orlando > Computer Engineering Major > University of Maryland -- Kyle Orlando Computer Engineering Major University of Maryland -- Claudio Martella [email protected]<mailto:[email protected]> -- Kyle Orlando Computer Engineering Major University of Maryland -- Claudio Martella [email protected]<mailto:[email protected]> -- Kyle Orlando Computer Engineering Major University of Maryland
