Re: Compiling Giraph 1.1

Hi Ryan, i'm facing same problem here. Did you find any solution for this? I'm building giraph with another hadoop version, but the problem appears to be the same, that dependency can't be resolved: mvn -e -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests clean install I'm not applying the fix of

Re: Compiling Giraph 1.1

giraph And the build finish without errors, finally!! Good bye! 2015-02-22 23:31 GMT-03:00 José Luis Larroque larroques...@gmail.com: Hi Ryan, i'm facing same problem here. Did you find any solution for this? I'm building giraph with another hadoop version, but the problem appears

Reduce stuck at 0% in PageRankBenchmark - Giraph examples

Hi guys, i'm testing my Single Node Cluster Giraph installation using PageRankBenchmark example, in the following way: *$HADOOP_HOME/bin/hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/giraph-examples-1.1.0-for-hadoop-2.4.0-jar-with-dependencies.jar

OutOfMemory error while reading byes from edges in yarn

I'm doing a BFS algorithm in yarn, and i make a custom value for the data on my vertex (Vertex Data). But, after i did this, something went wrong for the process of reading edges. I trace the error to the following lines of code: - In ByteArrayEdges, the variable **serializedEdgesBytesUsed**

Re: Graph 1.1.0 - Worker failed during input split (currently not supported)

to check in your ~/*.ssh/config* - if you have one. Hope that helps :) Sonja Am 11.08.2015 um 03:46 schrieb José Luis Larroque: Hello everybody. I'm executing a BFS algoritm on giraph, and i hit with this error. I searched for uppercase letters in the following files: /etc/*hostname

Re: Problems to run org.apache.giraph.examples.SimpleShortestPathsComputation

Hi again, Roberto! I used to build it with this line: mvn package -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests. I don't remember the exact solution that i took, but you have two options as far as i know: - Make that method public, by editing those clases manually - Remove the

Re: Giraph 1.2 fails to find org.apache.giraph:giraph-rexster-io:jar:1.2.0-SNAPSHOT

I'm not an expert, but i faced the same problem a time ago. Here are the steps that i did for solving it. I hope that could be useful for you too, here they are: *Don't use the "clone" command indicated in the "Quick Start Guide"*. The pom has *1.2* version, so is looking for dependencies for

Correct use of Writable interface - Vertex with Id text and a complex value as value (be worth of redundancy)

Hi, i'm Jose, and i'm doing a program that could read a text file in Giraph like this: Portada1.0Sugerencia1.0 Sugerencia1.0Portada1.0 It has this form: VertexId VertexValue EdgeId EdgeValue And all of those are separated for a TAB (/t). For reading this, i implemented

Re: OutOfMemory error while reading byes from edges in yarn

This was a memory configuration issue, after i fix it, the real problem was another (i will send a different email with it) Thanks! Jose 2015-08-20 11:38 GMT-03:00 José Luis Larroque <larroques...@gmail.com>: > I'm doing a BFS algorithm in yarn, and i make a custom value for the dat

Re: java.io.IOException: ensureRemaining: Only * bytes remaining, trying to read *

way ;) 2015-09-08 20:01 GMT-03:00 José Luis Larroque <larroques...@gmail.com>: > Hi guys, i'm having some problems with custom classes in giraph. I made a > VertexInput and Output format, but i always getting the following error: > > *java.io.IOException: ensureRemaining: Only

Re: Why failed jobs do not exit !

Hi Khaled, When you don't have enough memory and giraph try to create a container and fails, it will try to do it again and again, until it can or until it reach the maximum attemps, for example. In my case, for solving this problem i have to modify several configurations files in my cluster in

Re: Master compute and workers

I believe that you are seeing this from a weird perspective. Personal opinion's aside, maybe reading pregel's paper could be of great benefit, because Giraph is based on it. Regarding to your questions: Workers stop when all* vertex* are halt, and every worker has a partition of vertex.

How long a Message can be in Giraph

Hi all ! I'm trying to determinate how long a Message can be in Giraph. I checked the Text implementation, and inside there is a byte array. So, according to questions from SO, the maximum size of a message would be *around

Re: How long a Message can be in Giraph

Hi again, Being more accurate, how long can be the string sended, if the message is of Type Text. Any help would be greatly appreciated. Bye! Jose 2016-03-14 23:42 GMT-03:00 José Luis Larroque <larroques...@gmail.com>: > Hi all ! > > I'm trying to determinate how lon

Container gets killed on Worker, when doing flush, after completing superstep, and the entire application hangs

Hi all !! I know that all are very busy with giraph 1.2.0, but i hope that someone can help me :) I'm running a **Giraph** application on **EMR**. I'm using a cluster of 1 master and 10 slaves, all **m3.2xlarge** machines. The application consist, basically, on a BFS through the spanish

Re: Stop Giraph application when reach certain amount of time

I finally make a bash script that waits an amount of time before killing any app :( -- *José Luis Larroque* Analista Programador Universitario - Facultad de Informática - UNLP Desarrollador Java en LIFIA 2017-01-26 23:32 GMT-03:00 José Luis Larroque <larroques...@gmail.com>: >

Re: Stop Giraph application when reach certain amount of time

workaround for this problem? Any help will be greatly appreciated. bye! José -- *José Luis Larroque* Analista Programador Universitario - Facultad de Informática - UNLP Desarrollador Java en LIFIA 2017-01-25 22:51 GMT-03:00 José Luis Larroque <larroques...@gmail.com>: > Sorry,

Re: Stop Giraph application when reach certain amount of time

that took 12 seconds and the entire Giraph application doesn't get stopped. Thanks in advance! -- *José Luis Larroque* Analista Programador Universitario - Facultad de Informática - UNLP Desarrollador Java en LIFIA 2017-01-25 1:33 GMT-03:00 Sergey Edunov <edu...@gmail.com>: > H

Stop Giraph application when reach certain amount of time

JobTimeMilliseconds* -> With LOG level in DEBUG, i couldn't see any impact of using this option. But yet, i'm not getting the expected result, and i have Giraph applications that take like 12000 seconds or more (a big waste of time, resources and money). Any help will be greatly appreciated. T

Re: Problem Running Giraph application in a cluster

Great ! -- *José Luis Larroque* Analista Programador Universitario - Facultad de Informática - UNLP Desarrollador Java y .NET en LIFIA 2017-02-20 6:35 GMT-03:00 Sai Ganesh Muthuraman <saiganesh...@gmail.com>: > Hi Jose, > > Thanks for that, I used *javac -source 1.7 -target 1.7

Re: Problem Running Giraph application in a cluster

alues like this one: https://www.mail-archive.com/user@giraph.apache.org/msg02628.html Bye -- *José Luis Larroque* Analista Programador Universitario - Facultad de Informática - UNLP Desarrollador Java y .NET en LIFIA 2017-02-16 7:32 GMT-03:00 Sai Ganesh Muthuraman <saiganesh...@gmail.com&g

Re: RELATION BETWEEN THE NUMBER OF GIRAPH WORKERS AND THE PROBLEM SIZE

and over. -- *José Luis Larroque* Analista Programador Universitario - Facultad de Informática - UNLP Desarrollador Java y .NET en LIFIA 2017-02-25 12:38 GMT-03:00 Sai Ganesh Muthuraman <saiganesh...@gmail.com>: > Hi Jose, > > Which logs do I have to look into exactly, because in

Re: RELATION BETWEEN THE NUMBER OF GIRAPH WORKERS AND THE PROBLEM SIZE

don't have checkpointing enabled apparently, so the entire job is being killed. I reccomend that you look in your containers logs and try to detect why one or more workers are dying when you have bigger files. Bye! -- *José Luis Larroque* Analista Programador Universitario - Facultad de Informát

Re: RELATION BETWEEN THE NUMBER OF GIRAPH WORKERS AND THE PROBLEM SIZE

I remember that a good practice is using 1 worker per node, there are several emails recommending this. It's the best way to use the maximum RAM available in the cluster i believe. Bye -- *José Luis Larroque* Analista Programador Universitario - Facultad de Informática - UNLP Desarrollador

Re: Problem Running Giraph application in a cluster

-- *José Luis Larroque* Analista Programador Universitario - Facultad de Informática - UNLP Desarrollador Java y .NET en LIFIA 2017-02-17 2:56 GMT-03:00 Sai Ganesh Muthuraman <saiganesh...@gmail.com>: > Hi Jose, > > In fact, this is the running status of the application > &

Re: RELATION BETWEEN THE NUMBER OF GIRAPH WORKERS AND THE PROBLEM SIZE

your memory problems. At least until you have a clue of what is going on. - You should be using Giraph 1.2 that has a better support for out of core, instead of the previous 1.1 version. Are you using it? Check this tips and tell us any new information. Bye -- *José Luis Larroque* Analista

Re: RELATION BETWEEN THE NUMBER OF GIRAPH WORKERS AND THE PROBLEM SIZE

Another link with help: http://stackoverflow.com/questions/32713587/how-to-keep-yarns-log-files. Maybe you could test the algorithm locally instead of running it on the cluster, for a better understanding of the relation between yarn and Giraph. Bye -- *José Luis Larroque* Analista Programador

Re: Graph Data

Hi agron, here is a link that maybe can help you. http://stackoverflow.com/questions/30124671/import-save-snap-facebook-edges-in-apache-giraph?rq=1 Bye! Jose 2016-08-28 7:24 GMT-03:00 agc studio : > Hi all, > > I was wondering if there are any sample graph data from

Re: Lease renewer daemon for [] with renew id 1 exited - HDFS Issue regarding Giraph?

should i do. Any help? Bye Jose 2016-08-29 23:46 GMT-03:00 José Luis Larroque <larroques...@gmail.com>: > Hi guys, > > I have an application that gives me error. > > Lease renewer daemon for [] with renew id 1 exited > > In the superstep 2 of a Giraph Applicat

Lease renewer daemon for [] with renew id 1 exited - HDFS Issue regarding Giraph?

Hi guys, I have an application that gives me error. Lease renewer daemon for [] with renew id 1 exited In the superstep 2 of a Giraph Application. For some reason, after finishing a superstep, the worker don't do much, only gives this as output: 16/08/30 00:48:35 INFO netty.NettyClient:

Giraph application get stuck, on superstep 4, all workers active but without progress

Hi again guys! I'm doing BFS search through the Wikipedia (spanish edition) site. I converted the dump ( https://dumps.wikimedia.org/eswiki/20160601) into a file that could be read with Giraph. The BFS is searching for paths, and its all ok until

Re: Giraph application get stuck, on superstep 4, all workers active but without progress

, but still, same final results. Please, help. 2016-08-26 21:24 GMT-03:00 José Luis Larroque <larroques...@gmail.com>: > Hi again guys! > > I'm doing BFS search through the Wikipedia (spanish edition) site. I > converted the dump <https://dumps.wikimedia.org/eswiki

Re: giraph.numInputThreads execution time for "input superstep" it's the same using 1 or 8 threads, how this can be possible?

t; > On Thu, Aug 25, 2016 at 5:36 PM, José Luis Larroque < > larroques...@gmail.com> wrote: > >> he cluster used for this was 1 master and one slave, both of a r3.8xlarge >> EC2 instance on AWS. >> >> 2016-08-25 19:26 GMT-03:00 José Luis Larroque <larroque

Re: giraph.numInputThreads execution time for "input superstep" it's the same using 1 or 8 threads, how this can be possible?

n get benefit from parallelism reading multiple input splits at > the same time, which reduces the lock contention on each partition while > reading). > > On Fri, Aug 26, 2016 at 6:52 AM, José Luis Larroque < > larroques...@gmail.com> wrote: > >> Hi Hassan, thanks for your

Re: Giraph application get stuck, on superstep 4, all workers active but without progress

of each message is taking to long, and i have the entire cluster waiting for it. I Know that there are other tecnologies por post-processing results, maybe i should use one of them? Bye! Jose 2016-08-27 21:33 GMT-03:00 José Luis Larroque <larroques...@gmail.com>: > Using giraph.maxNumberOfOpe

Re: Giraph application get stuck, on superstep 4, all workers active but without progress

Problem solved, i optimize the proccesing of each message, and i could solve it. Sorry for the spam guys :D Bye! Jose 2016-08-28 15:23 GMT-03:00 José Luis Larroque <larroques...@gmail.com>: > Ok, i understand what is happening now. > > I starting to use more compute threads, bec

Combiners performance

Hi guys, i recently implement a MessageCombiner, and testing in an application that previosly work, it takes siginficant more time. Is there a way to paralelize its work? As for example, giraph.numCombinerThreads or something like that. Sorry if the question is too dumb. Thanks in advance! Jose

Re: Combiners performance

I found something: giraph.async.message.store (false by default) that will enable async messaging giraph.async.message.store.threads But i'm not sure if those options can work together with a Combiner. Thanks! Jose 2016-08-28 22:00 GMT-03:00 José Luis Larroque <larroques...@gmail.com>:

Re: giraph.numInputThreads execution time for "input superstep" it's the same using 1 or 8 threads, how this can be possible?

he cluster used for this was 1 master and one slave, both of a r3.8xlarge EC2 instance on AWS. 2016-08-25 19:26 GMT-03:00 José Luis Larroque <larroques...@gmail.com>: > I'm doing BFS search through the Wikipedia (spanish edition) site. I > converted the [dump][1] into a file that c

giraph.numInputThreads execution time for "input superstep" it's the same using 1 or 8 threads, how this can be possible?

I'm doing BFS search through the Wikipedia (spanish edition) site. I converted the [dump][1] into a file that could be read with Giraph. Using 1 worker, a file of 1 GB took 492 seconds. I executed Giraph with this command: /home/hadoop/bin/yarn jar /home/hadoop/giraph/giraph.jar

Re: Help for ruuning giraph example on multi-node cluster

You should check your memory settings. Apparently, for some reason the right amount of containers can't be created, so your applicaction won't make any progress until you fix your configuration. -- *José Luis Larroque* Analista Programador Universitario - Facultad de Informática - UNLP

Re: RELATION BETWEEN THE NUMBER OF GIRAPH WORKERS AND THE PROBLEM SIZE

i'm unable to help you in this issue.. -- *José Luis Larroque* Analista Programador Universitario - Facultad de Informática - UNLP Desarrollador Java y .NET en LIFIA 2017-03-02 14:18 GMT-03:00 Sai Ganesh Muthuraman <saiganesh...@gmail.com>: > Hi, > > I tried building giraph-