[jira] [Updated] (HAMA-732) Investigate the usage of Apache DirectMemory to store vertices
[ https://issues.apache.org/jira/browse/HAMA-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommaso Teofili updated HAMA-732: - Attachment: HAMA-732.3.patch last version of the patch with DM 0.2 (released yesterday) Investigate the usage of Apache DirectMemory to store vertices -- Key: HAMA-732 URL: https://issues.apache.org/jira/browse/HAMA-732 Project: Hama Issue Type: Improvement Components: graph Affects Versions: 0.6.0 Reporter: Tommaso Teofili Assignee: Tommaso Teofili Priority: Minor Fix For: 0.7.0 Attachments: HAMA-732.2.patch, HAMA-732.3.patch, HAMA-732.patch Since Apache DirectMemory can keep objects outside of the java heap that could help on improving performances. I created a simple extension implementation of VerticesInfo which uses Apache DirectMemory CacheService instead of a List, performance seem to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: svn commit: r1523425 - in /hama/trunk/c++: pom.xml src/main/native/pipes/impl/HamaPipes.cc
Sorry, I'm on vacation, will be back 2 days later.
--
Best Regards, Edward J. Yoon
@eddieyoon
On 2013. 9. 17., at 오후 5:40, Tommaso Teofili tommaso.teof...@gmail.com wrote:
I think we need Edward to run 'mvn release:rollback' as soon as possible
(as latest vote has been canceled) and then commit this again.
Tommaso
2013/9/15 chl...@apache.org
Author: chl501
Date: Sun Sep 15 10:20:01 2013
New Revision: 1523425
URL: http://svn.apache.org/r1523425
Log:
HAMA-802: Skip Hama Pipes native build when cmake is missing
Modified:
hama/trunk/c++/pom.xml
hama/trunk/c++/src/main/native/pipes/impl/HamaPipes.cc
Modified: hama/trunk/c++/pom.xml
URL:
http://svn.apache.org/viewvc/hama/trunk/c%2B%2B/pom.xml?rev=1523425r1=1523424r2=1523425view=diff
==
--- hama/trunk/c++/pom.xml (original)
+++ hama/trunk/c++/pom.xml Sun Sep 15 10:20:01 2013
@@ -31,7 +31,7 @@
descriptionApache Hama Pipes/description
packagingpom/packaging
-profiles
+profiles
profile
idnative/id
activation
@@ -49,16 +49,32 @@
goalsgoalrun/goal/goals
configuration
target
- mkdir dir=${project.build.directory}/native /
- exec executable=cmake
dir=${project.build.directory}/native failonerror=true
-arg line=${basedir}/src/
-DJVM_ARCH_DATA_MODEL=${sun.arch.data.model} /
- /exec
- exec executable=make
dir=${project.build.directory}/native failonerror=true
-arg line=VERBOSE=1 /
- /exec
- !-- The second make is a workaround for
HADOOP-9215. It can
- be removed when version 2.6 of cmake is no
longer supported . --
- exec executable=make
dir=${project.build.directory}/native failonerror=true /
+ taskdef
resource=net/sf/antcontrib/antcontrib.properties
classpathref=maven.plugin.classpath /
+ !-- Check if cmake is installed --
+ property environment=env /
+ if
+or
+ available file=cmake filepath=${env.PATH}
/
+ !-- on Windows it can be Path, path --
+ available file=cmake filepath=${env.Path}
/
+ available file=cmake filepath=${env.path}
/
+/or
+then
+ mkdir dir=${project.build.directory}/native
/
+ exec executable=cmake
dir=${project.build.directory}/native failonerror=true
+arg line=${basedir}/src/
-DJVM_ARCH_DATA_MODEL=${sun.arch.data.model} /
+ /exec
+ exec executable=make
dir=${project.build.directory}/native failonerror=true
+arg line=VERBOSE=1 /
+ /exec
+ !-- The second make is a workaround for
HADOOP-9215. It can
+ be removed when version 2.6 of cmake is no
longer supported . --
+ exec executable=make
dir=${project.build.directory}/native failonerror=true /
+/then
+else
+ echoCMAKE is missing! Skipping native
build!/echo
+/else
+ /if
/target
/configuration
/execution
@@ -75,6 +91,13 @@
/execution
--
/executions
+ dependencies
+dependency
+ groupIdant-contrib/groupId
+ artifactIdant-contrib/artifactId
+ version20020829/version
+/dependency
+ /dependencies
/plugin
/plugins
/build
Modified: hama/trunk/c++/src/main/native/pipes/impl/HamaPipes.cc
URL:
http://svn.apache.org/viewvc/hama/trunk/c%2B%2B/src/main/native/pipes/impl/HamaPipes.cc?rev=1523425r1=1523424r2=1523425view=diff
==
--- hama/trunk/c++/src/main/native/pipes/impl/HamaPipes.cc (original)
+++ hama/trunk/c++/src/main/native/pipes/impl/HamaPipes.cc Sun Sep 15
10:20:01 2013
@@ -23,6 +23,7 @@
#include map
#include vector
+#include unistd.h
#include errno.h
#include netinet/in.h
#include stdint.h
[jira] [Resolved] (HAMA-794) When aggregator is enabled, resending message in each message receiving round impacts the iteration performance.
[
https://issues.apache.org/jira/browse/HAMA-794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
MaoYuan Xian resolved HAMA-794.
---
Resolution: Duplicate
Fix Version/s: 0.7.0
Assignee: MaoYuan Xian
When aggregator is enabled, resending message in each message receiving round
impacts the iteration performance.
Key: HAMA-794
URL: https://issues.apache.org/jira/browse/HAMA-794
Project: Hama
Issue Type: Improvement
Components: graph
Reporter: MaoYuan Xian
Assignee: MaoYuan Xian
Fix For: 0.7.0
When aggregator is enabled, in each message receiving round, all received
message have been resent to put them behind the aggregate message.
doAggregationUpdates method of GraphJobRunner Class:
{code}
if (aggregationRunner.isEnabled() iteration 1) {
// in case we need to sync, we need to replay the messages that already
// are added to the queue. This prevents loosing messages when using
// aggregators.
if (firstVertexMessage != null) {
peer.send(peer.getPeerName(), firstVertexMessage);
}
GraphJobMessage msg = null;
while ((msg = peer.getCurrentMessage()) != null) {
peer.send(peer.getPeerName(), msg);
}
// now sync
peer.sync();
...
{code}
Should we do some improvement here? Record the original receiveQueue before
the sync, and after sync operation, read the aggregator updated value
firstly, then restore the original receiveQueue such that avoiding the
message resending operations.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Hama-Nightly-for-Hadoop-2.x #34
See https://builds.apache.org/job/Hama-Nightly-for-Hadoop-2.x/34/ -- [...truncated 478 lines...] A bin/zookeepers.sh A bin/hama-daemon.sh A bin/hama-config.sh A bin/hama-daemons.sh A bin/stop-bspd.sh A bin/hama A bin/start-bspd.sh A c++ A c++/src A c++/src/JNIFlags.cmake A c++/src/CMakeLists.txt A c++/src/main A c++/src/main/native A c++/src/main/native/pipes A c++/src/main/native/pipes/impl A c++/src/main/native/pipes/impl/HamaPipes.cc A c++/src/main/native/pipes/api A c++/src/main/native/pipes/api/hama A c++/src/main/native/pipes/api/hama/TemplateFactory.hh A c++/src/main/native/pipes/api/hama/Pipes.hh A c++/src/main/native/utils A c++/src/main/native/utils/impl A c++/src/main/native/utils/impl/StringUtils.cc A c++/src/main/native/utils/impl/SerialUtils.cc A c++/src/main/native/utils/api A c++/src/main/native/utils/api/hadoop A c++/src/main/native/utils/api/hadoop/Splitter.hh A c++/src/main/native/utils/api/hadoop/StringUtils.hh A c++/src/main/native/utils/api/hadoop/SerialUtils.hh A c++/src/main/native/utils/m4 A c++/pom.xml A ml A ml/pom.xml A ml/src A ml/src/test A ml/src/test/resources A ml/src/test/resources/logistic_regression_data.txt AUml/src/test/resources/vd_file_sample.txt A ml/src/test/resources/dimensional_reduction.txt A ml/src/test/resources/linear_regression_data.txt A ml/src/test/java A ml/src/test/java/org A ml/src/test/java/org/apache A ml/src/test/java/org/apache/hama A ml/src/test/java/org/apache/hama/ml A ml/src/test/java/org/apache/hama/ml/kmeans AUml/src/test/java/org/apache/hama/ml/kmeans/TestKMeansBSP.java A ml/src/test/java/org/apache/hama/ml/regression A ml/src/test/java/org/apache/hama/ml/regression/TestLinearRegression.java AU ml/src/test/java/org/apache/hama/ml/regression/LinearRegressionModelTest.java A ml/src/test/java/org/apache/hama/ml/regression/TestLogisticRegression.java AU ml/src/test/java/org/apache/hama/ml/regression/LogisticRegressionModelTest.java AU ml/src/test/java/org/apache/hama/ml/regression/VectorDoubleFileInputFormatTest.java A ml/src/test/java/org/apache/hama/ml/MLTestBase.java A ml/src/test/java/org/apache/hama/ml/perception A ml/src/test/java/org/apache/hama/ml/perception/TestSmallMultiLayerPerceptron.java A ml/src/test/java/org/apache/hama/ml/perception/TestSmallMLPMessage.java A ml/src/test/java/org/apache/hama/ml/math A ml/src/test/java/org/apache/hama/ml/math/TestDenseDoubleVector.java A ml/src/test/java/org/apache/hama/ml/math/TestFunctionFactory.java A ml/src/test/java/org/apache/hama/ml/math/TestDenseDoubleMatrix.java A ml/src/test/java/org/apache/hama/ml/ann A ml/src/test/java/org/apache/hama/ml/ann/TestSmallLayeredNeuralNetworkMessage.java A ml/src/test/java/org/apache/hama/ml/ann/TestAutoEncoder.java A ml/src/test/java/org/apache/hama/ml/ann/TestSmallLayeredNeuralNetwork.java A ml/src/main A ml/src/main/java A ml/src/main/java/org A ml/src/main/java/org/apache A ml/src/main/java/org/apache/hama A ml/src/main/java/org/apache/hama/ml A ml/src/main/java/org/apache/hama/ml/perception A ml/src/main/java/org/apache/hama/ml/perception/MultiLayerPerceptron.java A ml/src/main/java/org/apache/hama/ml/perception/MLPMessage.java A ml/src/main/java/org/apache/hama/ml/perception/SmallMultiLayerPerceptron.java A ml/src/main/java/org/apache/hama/ml/perception/PerceptronTrainer.java A ml/src/main/java/org/apache/hama/ml/perception/SmallMLPMessage.java A ml/src/main/java/org/apache/hama/ml/perception/SmallMLPTrainer.java A ml/src/main/java/org/apache/hama/ml/math A ml/src/main/java/org/apache/hama/ml/math/Function.java A ml/src/main/java/org/apache/hama/ml/math/Tuple.java A ml/src/main/java/org/apache/hama/ml/math/Tanh.java A ml/src/main/java/org/apache/hama/ml/math/Sigmoid.java A ml/src/main/java/org/apache/hama/ml/math/DoubleVector.java A ml/src/main/java/org/apache/hama/ml/math/DoubleDoubleVectorFunction.java A ml/src/main/java/org/apache/hama/ml/math/DoubleMatrix.java A ml/src/main/java/org/apache/hama/ml/math/IdentityFunction.java A ml/src/main/java/org/apache/hama/ml/math/DoubleVectorFunction.java A ml/src/main/java/org/apache/hama/ml/math/CrossEntropy.java A ml/src/main/java/org/apache/hama/ml/math/DoubleDoubleFunction.java A ml/src/main/java/org/apache/hama/ml/math/DenseDoubleVector.java
