Review Request: GIRAPH-144 GiraphJob should not extend Job (users should not be able to call Job methods like waitForCompletion or setMapper..etc)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3823/ --- Review request for giraph. Summary --- GiraphJob uses composition with Job instead of extending Job (exposing all Job methods to users is bad). This addresses bug GIRAPH-144. https://issues.apache.org/jira/browse/GIRAPH-144 Diffs - http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java 1241611 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/BspCase.java 1241611 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestJsonBase64Format.java 1241611 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleVertexWithWorkerContext.java 1241611 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java 1241611 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/GiraphRunner.java 1241611 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java 1241611 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleShortestPathsVertex.java 1241611 Diff: https://reviews.apache.org/r/3823/diff Testing --- local and MR unittests. Thanks, Avery
[jira] [Updated] (GIRAPH-145) Change partition request log level to debug rather than info
[
https://issues.apache.org/jira/browse/GIRAPH-145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated GIRAPH-145:
---
Attachment: GIRAPH-145.patch
Quick patch to go down to debug level. Verified with tests and cluster run.
Change partition request log level to debug rather than info
Key: GIRAPH-145
URL: https://issues.apache.org/jira/browse/GIRAPH-145
Project: Giraph
Issue Type: Improvement
Components: bsp
Affects Versions: 0.2.0
Reporter: Jakob Homan
Assignee: Jakob Homan
Fix For: 0.2.0
Attachments: GIRAPH-145.patch
{code:title=BasicRPCCommunications.java|borderStyle=solid}
if (LOG.isInfoEnabled()) {
LOG.info(sendPartitionReq: Sending to + rpcProxy.getName() +
+ addr + from + workerInfo +
, with partition + partition);
}{code}
is too chatty. We're seeing thousands and sounds of these lines for larger
graphs. This should be at debug level...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-145) Change partition request log level to debug rather than info
[
https://issues.apache.org/jira/browse/GIRAPH-145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13204887#comment-13204887
]
Claudio Martella commented on GIRAPH-145:
-
+1, go on.
Change partition request log level to debug rather than info
Key: GIRAPH-145
URL: https://issues.apache.org/jira/browse/GIRAPH-145
Project: Giraph
Issue Type: Improvement
Components: bsp
Affects Versions: 0.2.0
Reporter: Jakob Homan
Assignee: Jakob Homan
Fix For: 0.2.0
Attachments: GIRAPH-145.patch
{code:title=BasicRPCCommunications.java|borderStyle=solid}
if (LOG.isInfoEnabled()) {
LOG.info(sendPartitionReq: Sending to + rpcProxy.getName() +
+ addr + from + workerInfo +
, with partition + partition);
}{code}
is too chatty. We're seeing thousands and sounds of these lines for larger
graphs. This should be at debug level...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-142) _hadoopBsp should be prefixable via configuration
[ https://issues.apache.org/jira/browse/GIRAPH-142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated GIRAPH-142: --- Attachment: GIRAPH-142.patch Patch to add new config value, giraph.zkBaseZNode, that is the top-level for all giraph-created content on the ZK server. New unit test. Verified on running cluster as well. _hadoopBsp should be prefixable via configuration - Key: GIRAPH-142 URL: https://issues.apache.org/jira/browse/GIRAPH-142 Project: Giraph Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Jakob Homan Assignee: Jakob Homan Fix For: 0.2.0 Attachments: GIRAPH-142.patch In multitennant zookeeper clusters, it would be good to be able to specify the base directory that's created for the _hadoopBsp znodes. This would also fix the issue we have with creating that directory in the source root during tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-142) _hadoopBsp should be prefixable via configuration
[ https://issues.apache.org/jira/browse/GIRAPH-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13205047#comment-13205047 ] Avery Ching commented on GIRAPH-142: Looks fine, could we just add a check somewhere that the path must start with / and throw an exception explaining to the user the problem? _hadoopBsp should be prefixable via configuration - Key: GIRAPH-142 URL: https://issues.apache.org/jira/browse/GIRAPH-142 Project: Giraph Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Jakob Homan Assignee: Jakob Homan Fix For: 0.2.0 Attachments: GIRAPH-142.patch In multitennant zookeeper clusters, it would be good to be able to specify the base directory that's created for the _hadoopBsp znodes. This would also fix the issue we have with creating that directory in the source root during tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-146) Maven is running the tests twice during builds
[
https://issues.apache.org/jira/browse/GIRAPH-146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13205131#comment-13205131
]
Jakob Homan commented on GIRAPH-146:
From a run of {{mvn site:site}}. Other targets have this too.
{noformat}
grep -n -A 10 T E S T S huh.txt
152: T E S T S
153
154-Running org.apache.giraph.examples.SimpleShortestPathVertexTest
155-12/02/09 17:12:09 INFO server.ZooKeeperServerMain: Starting server
156-12/02/09 17:12:09 INFO server.ZooKeeperServer: Server
environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
157-12/02/09 17:12:09 INFO server.ZooKeeperServer: Server
environment:host.name=jhoman-mn.linkedin.biz
158-12/02/09 17:12:09 INFO server.ZooKeeperServer: Server
environment:java.version=1.6.0_22
159-12/02/09 17:12:09 INFO server.ZooKeeperServer: Server
environment:java.vendor=Apple Inc.
160-12/02/09 17:12:09 INFO server.ZooKeeperServer: Server
environment:java.home=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
161-12/02/09 17:12:09 INFO server.ZooKeeperServer: Server
environment:java.class.path=/Users/jhoman/huh/huh2/g142/target/test-classes:/Users/jhoman/huh/huh2/g142/target/classes:/Users/jhoman/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar:/Users/jhoman/.m2/repository/org/apache/hadoop/hadoop-core/0.20.203.0/hadoop-core-0.20.203.0.jar:/Users/jhoman/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/Users/jhoman/.m2/repository/commons-httpclient/commons-httpclient/3.0.1/commons-httpclient-3.0.1.jar:/Users/jhoman/.m2/repository/commons-logging/commons-logging/1.0.3/commons-logging-1.0.3.jar:/Users/jhoman/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/Users/jhoman/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar:/Users/jhoman/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/jhoman/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/Users/jhoman/.m2/repository/commons-lang/commons-lang/2.4/commons-lang-2.4.jar:/Users/jhoman/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/jhoman/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/Users/jhoman/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/jhoman/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar:/Users/jhoman/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar:/Users/jhoman/.m2/repository/org/mortbay/jetty/servlet-api/2.5-20081211/servlet-api-2.5-20081211.jar:/Users/jhoman/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/Users/jhoman/.m2/repository/tomcat/jasper-runtime/5.5.12/jasper-runtime-5.5.12.jar:/Users/jhoman/.m2/repository/tomcat/jasper-compiler/5.5.12/jasper-compiler-5.5.12.jar:/Users/jhoman/.m2/repository/org/mortbay/jetty/jsp-api-2.1/6.1.14/jsp-api-2.1-6.1.14.jar:/Users/jhoman/.m2/repository/org/mortbay/jetty/servlet-api-2.5/6.1.14/servlet-api-2.5-6.1.14.jar:/Users/jhoman/.m2/repository/org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar:/Users/jhoman/.m2/repository/ant/ant/1.6.5/ant-1.6.5.jar:/Users/jhoman/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/Users/jhoman/.m2/repository/net/java/dev/jets3t/jets3t/0.7.1/jets3t-0.7.1.jar:/Users/jhoman/.m2/repository/net/sf/kosmosfs/kfs/0.3/kfs-0.3.jar:/Users/jhoman/.m2/repository/hsqldb/hsqldb/1.8.0.10/hsqldb-1.8.0.10.jar:/Users/jhoman/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:/Users/jhoman/.m2/repository/org/eclipse/jdt/core/3.1.1/core-3.1.1.jar:/Users/jhoman/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.0/jackson-core-asl-1.8.0.jar:/Users/jhoman/.m2/repository/org/apache/mahout/mahout-collections/1.0/mahout-collections-1.0.jar:/Users/jhoman/.m2/repository/com/google/guava/guava/r09/guava-r09.jar:/Users/jhoman/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.0/jackson-mapper-asl-1.8.0.jar:/Users/jhoman/.m2/repository/org/apache/zookeeper/zookeeper/3.3.3/zookeeper-3.3.3.jar:/Users/jhoman/.m2/repository/log4j/log4j/1.2.15/log4j-1.2.15.jar:/Users/jhoman/.m2/repository/javax/mail/mail/1.4/mail-1.4.jar:/Users/jhoman/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Users/jhoman/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/Users/jhoman/.m2/repository/org/apache/commons/commons-io/1.3.2/commons-io-1.3.2.jar:/Users/jhoman/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/jhoman/.m2/repository/net/iharder/base64/2.3.8/base64-2.3.8.jar:/Users/jhoman/.m2/repository/org/json/json/20090211/json-20090211.jar:/Users/jhoman/.m2/repository/org/mockito/mockito-all/1.8.5/mockito-all-1.8.5.jar:
162-12/02/09 17:12:09 INFO server.ZooKeeperServer: Server
[jira] [Commented] (GIRAPH-146) Maven is running the tests twice during builds
[ https://issues.apache.org/jira/browse/GIRAPH-146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13205137#comment-13205137 ] Jakob Homan commented on GIRAPH-146: This might be a hiccup on my side. The double run from site is to generate the test coverage data, and I can't get a second run now on package. I'll keep poking it. Maven is running the tests twice during builds -- Key: GIRAPH-146 URL: https://issues.apache.org/jira/browse/GIRAPH-146 Project: Giraph Issue Type: Bug Components: build Reporter: Jakob Homan I had a feeling the build time had jumped significantly... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
