[jira] [Updated] (GIRAPH-181) Add Hadoop 1.0 profile to pom.xml
[ https://issues.apache.org/jira/browse/GIRAPH-181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-181: - Attachment: GIRAPH-181.patch Add Hadoop 1.0 profile to pom.xml - Key: GIRAPH-181 URL: https://issues.apache.org/jira/browse/GIRAPH-181 Project: Giraph Issue Type: Improvement Components: build Affects Versions: 0.2.0 Reporter: Eugene Koontz Assignee: Eugene Koontz Fix For: 0.2.0 Attachments: GIRAPH-181.patch Hadoop 1.0.x is now considered the current stable version of Hadoop, according to http://hadoop.apache.org/common/releases.html#Download . This JIRA is to add support within Giraph's maven profile for the 1.0.x Hadoop release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-181) Add Hadoop 1.0 profile to pom.xml
[ https://issues.apache.org/jira/browse/GIRAPH-181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-181: - Attachment: GIRAPH-181.patch Add support for Hadoop 1.0.2 to README. Thanks for the reminder, Avery. Also added some whitespace formatting for consistency. Add Hadoop 1.0 profile to pom.xml - Key: GIRAPH-181 URL: https://issues.apache.org/jira/browse/GIRAPH-181 Project: Giraph Issue Type: Improvement Components: build Affects Versions: 0.2.0 Reporter: Eugene Koontz Assignee: Eugene Koontz Fix For: 0.2.0 Attachments: GIRAPH-181.patch, GIRAPH-181.patch Hadoop 1.0.x is now considered the current stable version of Hadoop, according to http://hadoop.apache.org/common/releases.html#Download . This JIRA is to add support within Giraph's maven profile for the 1.0.x Hadoop release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-168) Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than HADOOP_FACEBOOK) and remove usage of HADOOP
[
https://issues.apache.org/jira/browse/GIRAPH-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-168:
-
Attachment: GIRAPH-168.patch
-removes unneeded org.apache.hadoop.giraph.zkJar from Facebook profile
-additional README content regarding maven profile usage
{code}
mvn -Phadoop_non_secure clean verify
mvn -Phadoop_facebook
-Dhadoop.jar.path=/Users/ekoontz/hadoop-20/build/hadoop-0.20.1-dev-core.jar
clean verify
mvn -Phadoop_0.20.203 clean verify
mvn clean verify
mvn -Phadoop_0.23 clean verify
mvn -Phadoop_trunk clean verify
{code}
succeeds.
Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than
HADOOP_FACEBOOK) and remove usage of HADOOP
-
Key: GIRAPH-168
URL: https://issues.apache.org/jira/browse/GIRAPH-168
Project: Giraph
Issue Type: Improvement
Affects Versions: 0.2.0
Reporter: Eugene Koontz
Assignee: Eugene Koontz
Attachments: GIRAPH-168.patch, GIRAPH-168.patch, GIRAPH-168.patch,
GIRAPH-168.patch, GIRAPH-168.patch, GIRAPH-168.patch
This JIRA relates to the mail thread here:
http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser
Currently we check for the munge flags HADOOP, HADOOP_FACEBOOK and
HADOOP_NON_SECURE when using munge in a few places. Hopefully we can
eliminate usage of munge in the future, but until then, we can mitigate the
complexity by consolidating the number of flags checked. This JIRA renames
HADOOP_FACEBOOK to HADOOP_SECURE, and removes usages of HADOOP, to handle the
same conditional compilation requirements. It also makes it easier to add
more maven profiles so that we can easily increase our hadoop version
coverage.
This patch modifies the existing hadoop_facebook profile to use the new
HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.
It also adds a new hadoop maven profile, hadoop_trunk, which also sets
HADOOP_SECURE.
Finally, it adds a default profile, hadoop_0.20.203. This is needed so that
we can specify its dependencies separately from hadoop_trunk, because the
hadoop dependencies have changed between trunk and 0.205.0 - the former
requires hadoop-common, hadoop-mapreduce-client-core, and
hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.
With this patch, the following passes:
{code}
mvn clean verify mvn -Phadoop_trunk clean verify mvn -Phadoop_0.20.203
clean verify
{code}
Current problems:
* I left in place the usage of HADOOP_NON_SECURE, but note that the profile
that uses this is hadoop_non_secure, which fails to compile on trunk:
https://issues.apache.org/jira/browse/GIRAPH-167 .
* I couldn't get -Phadoop_facebook to work; does this work outside of
Facebook?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-171) total time in MasterThread.run() is calculated incorrectly
[
https://issues.apache.org/jira/browse/GIRAPH-171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-171:
-
Attachment: GIRAPH-171.patch
total time in MasterThread.run() is calculated incorrectly
--
Key: GIRAPH-171
URL: https://issues.apache.org/jira/browse/GIRAPH-171
Project: Giraph
Issue Type: Bug
Reporter: Eugene Koontz
Assignee: Eugene Koontz
Attachments: GIRAPH-171.patch
While running PageMarkBenchMark, I was seeing in the output:
{{graph.MasterThread(172): total: Took 1.3336739262910001E9 seconds.}}
This was because currently, in {{MasterThread.run()}}, we have:
{code}
LOG.info(total: Took +
((System.currentTimeMillis() / 1000.0d) -
setupSecs) + seconds.);
{code}
but it should be:
{code}
LOG.info(total: Took +
((System.currentTimeMillis() - startMillis) /
1000.0d) + seconds.);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-168) Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than HADOOP_FACEBOOK) and remove usage of HADOOP
[
https://issues.apache.org/jira/browse/GIRAPH-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-168:
-
Attachment: GIRAPH-168.patch
Latest patch flips the set of munge directives from {HADOOP_NEWRPC,
HADOOP_SECURE} to {HADOOP_OLDRPC,HADOOP_NON_SECURE}. HADOOP_NON_SECURE is a
flag used currently in trunk, so this is a return back to the current trunk
state.
Making old-RPC-signature and non-secure be the exceptional cases seems to me
better because if we remove older Hadoop versions, we'll have also removed the
need for having any munge directives.
Please see the flag/profile matrix for this patch below:
||profile||HADOOP_OLDRPC||HADOOP_NON_SECURE||
|hadoop_non_secure|x|x|
|hadoop_0.20.203|x||
|hadoop_0.23| | |
|hadoop_trunk| | |
|hadoop_facebook| | |
Simplify munge directive usage with new munge flag HADOOP_SECURE (rather than
HADOOP_FACEBOOK) and remove usage of HADOOP
-
Key: GIRAPH-168
URL: https://issues.apache.org/jira/browse/GIRAPH-168
Project: Giraph
Issue Type: Improvement
Affects Versions: 0.2.0
Reporter: Eugene Koontz
Assignee: Eugene Koontz
Attachments: GIRAPH-168.patch, GIRAPH-168.patch, GIRAPH-168.patch
This JIRA relates to the mail thread here:
http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser
Currently we check for the munge flags HADOOP, HADOOP_FACEBOOK and
HADOOP_NON_SECURE when using munge in a few places. Hopefully we can
eliminate usage of munge in the future, but until then, we can mitigate the
complexity by consolidating the number of flags checked. This JIRA renames
HADOOP_FACEBOOK to HADOOP_SECURE, and removes usages of HADOOP, to handle the
same conditional compilation requirements. It also makes it easier to add
more maven profiles so that we can easily increase our hadoop version
coverage.
This patch modifies the existing hadoop_facebook profile to use the new
HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.
It also adds a new hadoop maven profile, hadoop_trunk, which also sets
HADOOP_SECURE.
Finally, it adds a default profile, hadoop_0.20.203. This is needed so that
we can specify its dependencies separately from hadoop_trunk, because the
hadoop dependencies have changed between trunk and 0.205.0 - the former
requires hadoop-common, hadoop-mapreduce-client-core, and
hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.
With this patch, the following passes:
{code}
mvn clean verify mvn -Phadoop_trunk clean verify mvn -Phadoop_0.20.203
clean verify
{code}
Current problems:
* I left in place the usage of HADOOP_NON_SECURE, but note that the profile
that uses this is hadoop_non_secure, which fails to compile on trunk:
https://issues.apache.org/jira/browse/GIRAPH-167 .
* I couldn't get -Phadoop_facebook to work; does this work outside of
Facebook?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-167) mvn -Phadoop_non_secure clean verify fails
[
https://issues.apache.org/jira/browse/GIRAPH-167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-167:
-
Attachment: GIRAPH-167.patch
mvn -Phadoop_non_secure clean verify fails
--
Key: GIRAPH-167
URL: https://issues.apache.org/jira/browse/GIRAPH-167
Project: Giraph
Issue Type: Bug
Affects Versions: 0.2.0
Reporter: Eugene Koontz
Assignee: Eugene Koontz
Labels: build, hadoop
Attachments: GIRAPH-167.patch
The {{hadoop_non_secure}} profile, which uses hadoop 0.20.2, is failing to
compile:
{code}
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/Users/ekoontz/giraph/target/munged/main/org/apache/giraph/comm/RPCCommunications.java:[184,48]
cannot find symbol
symbol : variable versionID
location: class org.apache.giraph.comm.RPCCommunicationsI,V,E,M
[INFO] 1 error
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-162) BspCase.setup() should catch FileNotFoundException thrown from org.apache.hadoop.fs.FileSystem.listStatus()
[ https://issues.apache.org/jira/browse/GIRAPH-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-162: - Description: In hadoop trunk, org.apache.hadoop.fs.FileSystem.listStatus() is declared to throw both FileNotFoundException and IOException. The former (FileNotFoundException) is currently not caught when BspCase.setup() looks for the GiraphJob.ZOOKEEPER_MANAGER_DIR_DEFAULT directory in order to delete it. The listStatus() call throws FileNotException if this directory does not exist and causes several tests to fail when using Hadoop trunk. This exception should be caught and ignored during setup(), since it's not an error for this directory not to exist. (was: In hadoop trunk, org.apache.hadoop.fs.FileSystem.listStatus() is declared to throws both FileNotFoundException and IOException. The former (FileNotFoundException) is currently not caught when BspCase.setup() looks for the GiraphJob.ZOOKEEPER_MANAGER_DIR_DEFAULT directory in order to delete it. The listStatus() call throws FileNotException if this directory does not exist and causes several tests to fail when using Hadoop trunk. This exception should be caught and ignored during setup(), since it's not an error for this directory not to exist.) BspCase.setup() should catch FileNotFoundException thrown from org.apache.hadoop.fs.FileSystem.listStatus() --- Key: GIRAPH-162 URL: https://issues.apache.org/jira/browse/GIRAPH-162 Project: Giraph Issue Type: Bug Components: test Affects Versions: 0.2.0 Reporter: Eugene Koontz Fix For: 0.2.0 Attachments: GIRAPH-162.patch In hadoop trunk, org.apache.hadoop.fs.FileSystem.listStatus() is declared to throw both FileNotFoundException and IOException. The former (FileNotFoundException) is currently not caught when BspCase.setup() looks for the GiraphJob.ZOOKEEPER_MANAGER_DIR_DEFAULT directory in order to delete it. The listStatus() call throws FileNotException if this directory does not exist and causes several tests to fail when using Hadoop trunk. This exception should be caught and ignored during setup(), since it's not an error for this directory not to exist. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-164) fix 5 Line is longer than 80 characters style errors in GiraphRunner
[
https://issues.apache.org/jira/browse/GIRAPH-164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-164:
-
Attachment: GIRAPH-164.patch
fix 5 Line is longer than 80 characters style errors in GiraphRunner
--
Key: GIRAPH-164
URL: https://issues.apache.org/jira/browse/GIRAPH-164
Project: Giraph
Issue Type: Bug
Affects Versions: 0.2.0
Reporter: Eugene Koontz
Priority: Trivial
Fix For: 0.2.0
Attachments: GIRAPH-164.patch
{code}
file
name=/Users/ekoontz/giraph/src/main/java/org/apache/giraph/GiraphRunner.java
error line=155 severity=error message=Line is longer than 80
characters.
source=com.puppycrawl.tools.checkstyle.checks.sizes.LineLengthCheck/
error line=156 severity=error message=Line is longer than 80
characters.
source=com.puppycrawl.tools.checkstyle.checks.sizes.LineLengthCheck/
error line=158 severity=error message=Line is longer than 80
characters.
source=com.puppycrawl.tools.checkstyle.checks.sizes.LineLengthCheck/
error line=161 severity=error message=Line is longer than 80
characters.
source=com.puppycrawl.tools.checkstyle.checks.sizes.LineLengthCheck/
/file
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-165) checkstyle error: 'conf'hides a field' on line 154 of GraphRunner
[
https://issues.apache.org/jira/browse/GIRAPH-165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-165:
-
Attachment: GIRAPH-165.patch
checkstyle error: 'conf'hides a field' on line 154 of GraphRunner
-
Key: GIRAPH-165
URL: https://issues.apache.org/jira/browse/GIRAPH-165
Project: Giraph
Issue Type: Bug
Reporter: Eugene Koontz
Priority: Minor
Attachments: GIRAPH-165.patch
full checkstyle error is
{code}
file
name=/Users/ekoontz/giraph/src/main/java/org/apache/giraph/GiraphRunner.java
error line=154 column=21 severity=error message=apos;confapos;
hides a field.
source=com.puppycrawl.tools.checkstyle.checks.coding.HiddenFieldCheck/
/file
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-165) checkstyle error: 'conf' hides a field on line 154 of GraphRunner
[
https://issues.apache.org/jira/browse/GIRAPH-165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-165:
-
Summary: checkstyle error: 'conf' hides a field on line 154 of
GraphRunner (was: checkstyle error: 'conf'hides a field' on line 154 of
GraphRunner)
checkstyle error: 'conf' hides a field on line 154 of GraphRunner
Key: GIRAPH-165
URL: https://issues.apache.org/jira/browse/GIRAPH-165
Project: Giraph
Issue Type: Bug
Reporter: Eugene Koontz
Priority: Minor
Attachments: GIRAPH-165.patch
full checkstyle error is
{code}
file
name=/Users/ekoontz/giraph/src/main/java/org/apache/giraph/GiraphRunner.java
error line=154 column=21 severity=error message=apos;confapos;
hides a field.
source=com.puppycrawl.tools.checkstyle.checks.coding.HiddenFieldCheck/
/file
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-166) add '*.patch' to list of files that Apache Rat ignores
[ https://issues.apache.org/jira/browse/GIRAPH-166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-166: - Attachment: (was: pom.xml) add '*.patch' to list of files that Apache Rat ignores -- Key: GIRAPH-166 URL: https://issues.apache.org/jira/browse/GIRAPH-166 Project: Giraph Issue Type: Improvement Reporter: Eugene Koontz Priority: Trivial Attachments: GIRAPH-166.patch Apache Rat will complain about too many files without licenses if it finds any *.patch files in your working directory. Rat should ignore these since they are temp files that aren't included in the distribution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-166) add '*.patch' to list of files that Apache Rat ignores
[ https://issues.apache.org/jira/browse/GIRAPH-166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-166: - Attachment: GIRAPH-166.patch add '*.patch' to list of files that Apache Rat ignores -- Key: GIRAPH-166 URL: https://issues.apache.org/jira/browse/GIRAPH-166 Project: Giraph Issue Type: Improvement Reporter: Eugene Koontz Priority: Trivial Attachments: GIRAPH-166.patch Apache Rat will complain about too many files without licenses if it finds any *.patch files in your working directory. Rat should ignore these since they are temp files that aren't included in the distribution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-167) mvn -Phadoop_non_secure clean verify fails
[
https://issues.apache.org/jira/browse/GIRAPH-167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-167:
-
Description:
The {{hadoop_non_secure}} profile, which uses hadoop 0.20.2, is failing to
compile:
{code}
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/Users/ekoontz/giraph/target/munged/main/org/apache/giraph/comm/RPCCommunications.java:[184,48]
cannot find symbol
symbol : variable versionID
location: class org.apache.giraph.comm.RPCCommunicationsI,V,E,M
[INFO] 1 error
{code}
was:
The {{hadoop_non_secure}} profile, which uses hadoop 0.20.2, is failing to
compile:
{code}
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/Users/ekoontz/giraph/target/munged/main/org/apache/giraph/graph/partition/RangePartitionOwner.java:[26,27]
package org.apache.hadoop.io does not exist
[ERROR]
/Users/ekoontz/giraph/target/munged/main/org/apache/giraph/graph/partition/BasicPartitionOwner.java:[26,29]
package org.apache.hadoop.conf does not exist
[ERROR]
/Users/ekoontz/giraph/target/munged/main/org/apache/giraph/graph/partition/BasicPartitionOwner.java:[27,29]
package org.apache.hadoop.conf does not exist
[ERROR]
/Users/ekoontz/giraph/target/munged/main/org/apache/giraph/graph/partition/PartitionOwner.java:[22,27]
package org.apache.hadoop.io does not exist
[ERROR]
/Users/ekoontz/giraph/target/munged/main/org/apache/giraph/graph/partition/PartitionOwner.java:[27,40]
cannot find symbol
symbol: class Writable
{code}
(more error messages follow)
mvn -Phadoop_non_secure clean verify fails
--
Key: GIRAPH-167
URL: https://issues.apache.org/jira/browse/GIRAPH-167
Project: Giraph
Issue Type: Bug
Reporter: Eugene Koontz
The {{hadoop_non_secure}} profile, which uses hadoop 0.20.2, is failing to
compile:
{code}
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/Users/ekoontz/giraph/target/munged/main/org/apache/giraph/comm/RPCCommunications.java:[184,48]
cannot find symbol
symbol : variable versionID
location: class org.apache.giraph.comm.RPCCommunicationsI,V,E,M
[INFO] 1 error
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-168) Simplify munge directive usage with new munge flag HADOOP_SECURE rather than HADOOP_FACEBOOK and HADOOP_NON_SECURE
[
https://issues.apache.org/jira/browse/GIRAPH-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-168:
-
Attachment: GIRAPH-168.patch
Simplify munge directive usage with new munge flag HADOOP_SECURE rather than
HADOOP_FACEBOOK and HADOOP_NON_SECURE
--
Key: GIRAPH-168
URL: https://issues.apache.org/jira/browse/GIRAPH-168
Project: Giraph
Issue Type: Improvement
Affects Versions: 0.2.0
Reporter: Eugene Koontz
Assignee: Eugene Koontz
Attachments: GIRAPH-168.patch
This JIRA relates to the mail thread here:
http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser
Currently we check for the munge flags HADOOP and HADOOP_FACEBOOK and
HADOOP_NON_SECURE when using munge in a few places. Hopefully we can
eliminate usage of munge in the future, but until then, we can mitigate the
complexity by consolidating the number of flags checked. This JIRA proposes a
single flag, HADOOP_SECURE, to handle the same conditional compilation
requirements. It also makes it easier to add more maven profiles so that we
can easily increase our hadoop version coverage.
This patch modifies the existing hadoop_facebook profile to use the new
HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.
It also adds a new hadoop maven profile, hadoop_trunk, which also sets
HADOOP_SECURE.
Finally, it adds a default profile, hadoop_0.20.203. This is needed so that
we can specify its dependencies separately from hadoop_trunk, because the
hadoop dependencies have changed between trunk and 0.205.0 - the former
requires hadoop-common, hadoop-mapreduce-client-core, and
hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.
With this patch, the following passes:
{code}
mvn clean verify mvn -Phadoop_trunk clean verify mvn -Phadoop_0.20.203
clean verify
{code}
Current problems:
* I left in place the usage of HADOOP_NON_SECURE, but note that the profile
that uses this is hadoop_non_secure, which fails to compile on trunk:
https://issues.apache.org/jira/browse/GIRAPH-167 .
* I couldn't get -Phadoop_facebook to work; does this work outside of
Facebook?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-168) Simplify munge directive usage with new munge flag HADOOP_SECURE rather than HADOOP_FACEBOOK
[
https://issues.apache.org/jira/browse/GIRAPH-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koontz updated GIRAPH-168:
-
Summary: Simplify munge directive usage with new munge flag HADOOP_SECURE
rather than HADOOP_FACEBOOK (was: Simplify munge directive usage with new
munge flag HADOOP_SECURE rather than HADOOP_FACEBOOK and HADOOP_NON_SECURE)
Simplify munge directive usage with new munge flag HADOOP_SECURE rather than
HADOOP_FACEBOOK
Key: GIRAPH-168
URL: https://issues.apache.org/jira/browse/GIRAPH-168
Project: Giraph
Issue Type: Improvement
Affects Versions: 0.2.0
Reporter: Eugene Koontz
Assignee: Eugene Koontz
Attachments: GIRAPH-168.patch
This JIRA relates to the mail thread here:
http://mail-archives.apache.org/mod_mbox/incubator-giraph-dev/201203.mbox/browser
Currently we check for the munge flags HADOOP and HADOOP_FACEBOOK and
HADOOP_NON_SECURE when using munge in a few places. Hopefully we can
eliminate usage of munge in the future, but until then, we can mitigate the
complexity by consolidating the number of flags checked. This JIRA proposes a
single flag, HADOOP_SECURE, to handle the same conditional compilation
requirements. It also makes it easier to add more maven profiles so that we
can easily increase our hadoop version coverage.
This patch modifies the existing hadoop_facebook profile to use the new
HADOOP_SECURE munge flag, rather than HADOOP_FACEBOOK.
It also adds a new hadoop maven profile, hadoop_trunk, which also sets
HADOOP_SECURE.
Finally, it adds a default profile, hadoop_0.20.203. This is needed so that
we can specify its dependencies separately from hadoop_trunk, because the
hadoop dependencies have changed between trunk and 0.205.0 - the former
requires hadoop-common, hadoop-mapreduce-client-core, and
hadoop-mapreduce-client-common, whereas the latter requires hadoop-core.
With this patch, the following passes:
{code}
mvn clean verify mvn -Phadoop_trunk clean verify mvn -Phadoop_0.20.203
clean verify
{code}
Current problems:
* I left in place the usage of HADOOP_NON_SECURE, but note that the profile
that uses this is hadoop_non_secure, which fails to compile on trunk:
https://issues.apache.org/jira/browse/GIRAPH-167 .
* I couldn't get -Phadoop_facebook to work; does this work outside of
Facebook?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-158) Support YARN (next generation MapReduce)
[ https://issues.apache.org/jira/browse/GIRAPH-158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated GIRAPH-158: - Attachment: GIRAPH-158.patch This patch passes mvn verify with pom.xml modified to use 0.24.0-SNAPSHOT as hadoop.version. Support YARN (next generation MapReduce) Key: GIRAPH-158 URL: https://issues.apache.org/jira/browse/GIRAPH-158 Project: Giraph Issue Type: New Feature Reporter: Eugene Koontz Attachments: GIRAPH-158.patch YARN is a re-architecturing of the Hadoop MapReduce framework, described here: http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/YARN.html It would be good to offer support within Giraph for this framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
