[jira] [Commented] (HADOOP-8776) Provide an option in test-patch that can enable / disable compiling native code
[ https://issues.apache.org/jira/browse/HADOOP-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477351#comment-13477351 ] Jianbin Wei commented on HADOOP-8776: - I agree this needs to be done and merged into trunk. Another developer's time was wasted. :-) > Provide an option in test-patch that can enable / disable compiling native > code > --- > > Key: HADOOP-8776 > URL: https://issues.apache.org/jira/browse/HADOOP-8776 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0 >Reporter: Hemanth Yamijala >Assignee: Hemanth Yamijala >Priority: Minor > Attachments: HADOOP-8776.patch, HADOOP-8776.patch, HADOOP-8776.patch > > > The test-patch script in Hadoop source runs a native compile with the patch. > On platforms like MAC, there are issues with the native compile that make it > difficult to use test-patch. This JIRA is to try and provide an option to > make the native compilation optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7895) HADOOP_LOG_DIR has to be set explicitly when running from the tarball
[
https://issues.apache.org/jira/browse/HADOOP-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473937#comment-13473937
]
Jianbin Wei commented on HADOOP-7895:
-
Is this a duplication of HADOOP-8443
(https://issues.apache.org/jira/browse/HADOOP-8433)?
> HADOOP_LOG_DIR has to be set explicitly when running from the tarball
> -
>
> Key: HADOOP-7895
> URL: https://issues.apache.org/jira/browse/HADOOP-7895
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>
> When running bin and sbin commands from the tarball if HADOOP_LOG_DIR is not
> explicitly set in hadoop-env.sh it doesn't use HADOOP_HOME/logs by default
> like it used to, instead picks a wrong dir:
> {noformat}
> localhost: mkdir: cannot create directory `/eli': Permission denied
> localhost: chown: cannot access `/eli/eli': No such file or directory
> {noformat}
> We should have it default to HADOOP_HOME/logs or at least fail with a message
> if the dir doesn't exist, the env var isn't set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8608) Add Configuration API for parsing time durations
[
https://issues.apache.org/jira/browse/HADOOP-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469531#comment-13469531
]
Jianbin Wei commented on HADOOP-8608:
-
I am looking for some document like in TimeUnit.convert that includes examples
and warning. Not warning in the code.
{code}
/**
* Return time duration in the given time unit. Valid units are encoded in
* properties as suffixes: nanoseconds (ns), microseconds (us), milliseconds
* (ms), seconds (s), minutes (m), hours (h), and days (d).
* For example, the value can be 10ns, 10us, 10ms, 10s, and etc.
* Note getting time duration set in finer granularity as coarser granularity
can lose precision.
* For example, if property "example.duration" is set to 999ms,
* getTimeDuration("example.duration", 1L, TimeUnit.SECONDS) returns
0.
{code}
Some other minor points:
* In {{ParsedTimeDuration.unitFor}} the {{return null;}} is never executed.
* In the same method, you used "pdt". I guess you mean "ptd".
> Add Configuration API for parsing time durations
>
>
> Key: HADOOP-8608
> URL: https://issues.apache.org/jira/browse/HADOOP-8608
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
> Attachments: 8608-0.patch, 8608-1.patch
>
>
> Hadoop has a lot of configurations which specify durations or intervals of
> time. Unfortunately these different configurations have little consistency in
> units - eg some are in milliseconds, some in seconds, and some in minutes.
> This makes it difficult for users to configure, since they have to always
> refer back to docs to remember the unit for each property.
> The proposed solution is to add an API like {{Configuration.getTimeDuration}}
> which allows the user to specify the units with a postfix. For example,
> "10ms", "10s", "10m", "10h", or even "10d". For backwards-compatibility, if
> the user does not specify a unit, the API can specify the default unit, and
> warn the user that they should specify an explicit unit instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8608) Add Configuration API for parsing time durations
[
https://issues.apache.org/jira/browse/HADOOP-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469209#comment-13469209
]
Jianbin Wei commented on HADOOP-8608:
-
Comments as follows:
* I am kind of concerned about the "lose precision" from the conversion. That
is why I propose to give that control back to the caller to make the precision
lose explicitly. If we go this approach, at least I think we should document
this precision lose. Comments?
* The parsing of value is a bit loose. For example, it cannot handle "10S" or
"10 s". A strict format can reduce errors but may be inflexible and the
exception can be little harsh. Or we need to document the expecting format is
"10s" not "10 s" nor "10sec".
* Also the parsing part relies on the enum order implicitly. It works now.
But it may bite us later. A Pattern instead?
* It would be better to add the value of the timeduration property into LOG.
* Can you please change test from "testTime" to "testTimeDuration" for
consistency?
{code}
conf.setStrings("test.time.str", new String[]{"10S"});
assertEquals(1L, conf.getTimeDuration("test.time.str", 30,
MILLISECONDS));
{code}
It logs
2012-10-04 00:29:37,172 WARN conf.Configuration
(Configuration.java:getTimeDuration(1212)) - No unit for test.time.str assuming
MILLISECONDS
Better as "No unit for test.time.str (value: 10S), assuming MILLISECONDS" or
something like that with property value
> Add Configuration API for parsing time durations
>
>
> Key: HADOOP-8608
> URL: https://issues.apache.org/jira/browse/HADOOP-8608
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
> Attachments: 8608-0.patch
>
>
> Hadoop has a lot of configurations which specify durations or intervals of
> time. Unfortunately these different configurations have little consistency in
> units - eg some are in milliseconds, some in seconds, and some in minutes.
> This makes it difficult for users to configure, since they have to always
> refer back to docs to remember the unit for each property.
> The proposed solution is to add an API like {{Configuration.getTimeDuration}}
> which allows the user to specify the units with a postfix. For example,
> "10ms", "10s", "10m", "10h", or even "10d". For backwards-compatibility, if
> the user does not specify a unit, the API can specify the default unit, and
> warn the user that they should specify an explicit unit instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8776) Provide an option in test-patch that can enable / disable compiling native code
[
https://issues.apache.org/jira/browse/HADOOP-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468698#comment-13468698
]
Jianbin Wei commented on HADOOP-8776:
-
{quote}
Now test-patch will be platform specific. Ideally, it must be tested on all
platforms before committing.
{quote}
Yes I agree completely. As you said, the core is to fix it on "broken"
platforms. This is just a workaround.
{quote}
It has more logic (either --enable-native is passed OR it is a supported
platform, etc.). Not earth-shattering, but still needs to be understood in
context by someone fresh looking at it.
{quote}
I would say only the developer to fix the script will need to understand the
extra complexity. So it should be transparent to most other developers.
{quote}
The dependency it will put on us to have to track removing this once native
compile is fixed. Given that we cannot rely on contributors being always there
and watchful, I am worried that someone will forget to document in a related
JIRA to fix test-patch and we will end up in a situation that native compile is
fixed, and test-patch isn't testing it.
{quote}
The developer who disables the test needs to
* file a ticket to get "broken" platform fixed (if it is not there yet) AND
* document the needs to enable the test in the ticket. In this case if it
would be you :-)
Can you also print some messages to notify others that the native compilation
is disabled due to issues?
My goal is to provide best experiences for both end users and developers.
Other thoughts?
> Provide an option in test-patch that can enable / disable compiling native
> code
> ---
>
> Key: HADOOP-8776
> URL: https://issues.apache.org/jira/browse/HADOOP-8776
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0
>Reporter: Hemanth Yamijala
>Assignee: Hemanth Yamijala
>Priority: Minor
> Attachments: HADOOP-8776.patch, HADOOP-8776.patch, HADOOP-8776.patch
>
>
> The test-patch script in Hadoop source runs a native compile with the patch.
> On platforms like MAC, there are issues with the native compile that make it
> difficult to use test-patch. This JIRA is to try and provide an option to
> make the native compilation optional.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8608) Add Configuration API for parsing time durations
[
https://issues.apache.org/jira/browse/HADOOP-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468372#comment-13468372
]
Jianbin Wei commented on HADOOP-8608:
-
I am thinking the API can be
{code}
// return the property value in millisecond. The property value can be
specified as "10ms", "10s", "10m", "10h", or "10d"
public long getTimeDurationInMillisecond(String name, long defaultValue);
{code}
After that developers can use methods like
{code}
java.util.concurrent.TimeUnit.MILLISECONDS.toSeconds(long duration)
{code}
to convert to duration in second or other time units.
However, the conversion from TimeUnit may lost precision, e.g., 70 seconds will
be converted to 1 minute.
Any thoughts?
> Add Configuration API for parsing time durations
>
>
> Key: HADOOP-8608
> URL: https://issues.apache.org/jira/browse/HADOOP-8608
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>
> Hadoop has a lot of configurations which specify durations or intervals of
> time. Unfortunately these different configurations have little consistency in
> units - eg some are in milliseconds, some in seconds, and some in minutes.
> This makes it difficult for users to configure, since they have to always
> refer back to docs to remember the unit for each property.
> The proposed solution is to add an API like {{Configuration.getTimeDuration}}
> which allows the user to specify the units with a postfix. For example,
> "10ms", "10s", "10m", "10h", or even "10d". For backwards-compatibility, if
> the user does not specify a unit, the API can specify the default unit, and
> warn the user that they should specify an explicit unit instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8608) Add Configuration API for parsing time durations
[
https://issues.apache.org/jira/browse/HADOOP-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianbin Wei updated HADOOP-8608:
Description:
Hadoop has a lot of configurations which specify durations or intervals of
time. Unfortunately these different configurations have little consistency in
units - eg some are in milliseconds, some in seconds, and some in minutes. This
makes it difficult for users to configure, since they have to always refer back
to docs to remember the unit for each property.
The proposed solution is to add an API like {{Configuration.getTimeDuration}}
which allows the user to specify the units with a postfix. For example, "10ms",
"10s", "10m", "10h", or even "10d". For backwards-compatibility, if the user
does not specify a unit, the API can specify the default unit, and warn the
user that they should specify an explicit unit instead.
was:
Hadoop has a lot of configurations which specify durations or intervals of
time. Unfortunately these different configurations have little consistency in
units - eg some are in milliseconds, some in seconds, and some in minutes. This
makes it difficult for users to configure, since they have to always refer back
to docs to remember the unit for each property.
The proposed solution is to add an API like {{Configuration.getTimeDuration}}
which allows the user to specify the units with a prefix. For example, "10ms",
"10s", "10m", "10h", or even "10d". For backwards-compatibility, if the user
does not specify a unit, the API can specify the default unit, and warn the
user that they should specify an explicit unit instead.
> Add Configuration API for parsing time durations
>
>
> Key: HADOOP-8608
> URL: https://issues.apache.org/jira/browse/HADOOP-8608
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>
> Hadoop has a lot of configurations which specify durations or intervals of
> time. Unfortunately these different configurations have little consistency in
> units - eg some are in milliseconds, some in seconds, and some in minutes.
> This makes it difficult for users to configure, since they have to always
> refer back to docs to remember the unit for each property.
> The proposed solution is to add an API like {{Configuration.getTimeDuration}}
> which allows the user to specify the units with a postfix. For example,
> "10ms", "10s", "10m", "10h", or even "10d". For backwards-compatibility, if
> the user does not specify a unit, the API can specify the default unit, and
> warn the user that they should specify an explicit unit instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8776) Provide an option in test-patch that can enable / disable compiling native code
[
https://issues.apache.org/jira/browse/HADOOP-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467005#comment-13467005
]
Jianbin Wei commented on HADOOP-8776:
-
In hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh the platform
check is done through
{quote}
MAC_OSX=false
case "`uname`" in
Darwin*) MAC_OSX=true;;
esac
if $MAC_OSX; then
export HADOOP_OPTS="$HADOOP_OPTS -Djava.security.krb5.realm=
-Djava.security.krb5.kdc="
fi
{quote}
> Provide an option in test-patch that can enable / disable compiling native
> code
> ---
>
> Key: HADOOP-8776
> URL: https://issues.apache.org/jira/browse/HADOOP-8776
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0
>Reporter: Hemanth Yamijala
>Assignee: Hemanth Yamijala
>Priority: Minor
> Attachments: HADOOP-8776.patch, HADOOP-8776.patch, HADOOP-8776.patch
>
>
> The test-patch script in Hadoop source runs a native compile with the patch.
> On platforms like MAC, there are issues with the native compile that make it
> difficult to use test-patch. This JIRA is to try and provide an option to
> make the native compilation optional.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8776) Provide an option in test-patch that can enable / disable compiling native code
[ https://issues.apache.org/jira/browse/HADOOP-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466930#comment-13466930 ] Jianbin Wei commented on HADOOP-8776: - Hemanth, I agree that the core issue is the broken native compilation and that should be fixed. My suggestion is to disable the native compilation for "all" broken platforms by default, file tickets to have them fixed and enable native compilation as part of the fix. My purpose is to make it more convenient/efficient for other developers. Every time I generate a patch, I do the patch test and it fails due to javac warning. I had to check if it is because of native compilation or not. The whole process for compilation and checking takes about 3 minutes. If we can save this 3 minutes for each patch/developer, we may easily save thousands of minutes before the native compilation is fixed. This should justify a slightly more complicated workaround that may take you 60 minutes. Just my 2 cents. > Provide an option in test-patch that can enable / disable compiling native > code > --- > > Key: HADOOP-8776 > URL: https://issues.apache.org/jira/browse/HADOOP-8776 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0 >Reporter: Hemanth Yamijala >Assignee: Hemanth Yamijala >Priority: Minor > Attachments: HADOOP-8776.patch, HADOOP-8776.patch, HADOOP-8776.patch > > > The test-patch script in Hadoop source runs a native compile with the patch. > On platforms like MAC, there are issues with the native compile that make it > difficult to use test-patch. This JIRA is to try and provide an option to > make the native compilation optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8865) log warn when loading deprecated properties
[ https://issues.apache.org/jira/browse/HADOOP-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466423#comment-13466423 ] Jianbin Wei commented on HADOOP-8865: - Harsh, Do you have any idea why the org.apache.hadoop.ha.TestZKFailoverController fails due to timeout? I saw that the unit test also failed in HADOOP-8386 but the patch is accepted. Should the tests be disabled? https://issues.apache.org/jira/browse/HADOOP-8386?focusedCommentId=13466016&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13466016 Thanks. Jianbin > log warn when loading deprecated properties > --- > > Key: HADOOP-8865 > URL: https://issues.apache.org/jira/browse/HADOOP-8865 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.0.0 >Reporter: Jianbin Wei > Fix For: 3.0.0 > > Attachments: HADOOP-8865.patch, HADOOP-8865.patch, HADOOP-8865.patch > > > We need to log warn when loading deprecated properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8426) API to get info for deprecated key
[
https://issues.apache.org/jira/browse/HADOOP-8426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466418#comment-13466418
]
Jianbin Wei commented on HADOOP-8426:
-
{quote}
Using this API the user can judge in runtime to use the old key or new key.
{quote}
In which case that a user wants to use deprecated key?
> API to get info for deprecated key
> --
>
> Key: HADOOP-8426
> URL: https://issues.apache.org/jira/browse/HADOOP-8426
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Affects Versions: 0.23.1
>Reporter: Subroto Sanyal
>
> The current version of Hadoop deprecates many keys but, takes care of adding
> the new keys to the configuration accordingly.
> For the end user only logs are there which indicates the key is deprecated
> and the message also suggests the new key to be used.
> I was thinking; there should be an API/Utility which could probably provide
> the new key information when called with old key.
> Using this API the user can judge in runtime to use the old key or new key.
> Currently the Configuration provides an API only to check whether a key is
> deprecated or not but, doesn't provides a way to get the corresponding new
> key.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8865) log warn when loading deprecated properties
[ https://issues.apache.org/jira/browse/HADOOP-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8865: Status: Patch Available (was: Open) > log warn when loading deprecated properties > --- > > Key: HADOOP-8865 > URL: https://issues.apache.org/jira/browse/HADOOP-8865 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.0.0 >Reporter: Jianbin Wei > Fix For: 3.0.0 > > Attachments: HADOOP-8865.patch, HADOOP-8865.patch, HADOOP-8865.patch > > > We need to log warn when loading deprecated properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8865) log warn when loading deprecated properties
[ https://issues.apache.org/jira/browse/HADOOP-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8865: Attachment: HADOOP-8865.patch > log warn when loading deprecated properties > --- > > Key: HADOOP-8865 > URL: https://issues.apache.org/jira/browse/HADOOP-8865 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.0.0 >Reporter: Jianbin Wei > Fix For: 3.0.0 > > Attachments: HADOOP-8865.patch, HADOOP-8865.patch, HADOOP-8865.patch > > > We need to log warn when loading deprecated properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8865) log warn when loading deprecated properties
[ https://issues.apache.org/jira/browse/HADOOP-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8865: Status: Open (was: Patch Available) > log warn when loading deprecated properties > --- > > Key: HADOOP-8865 > URL: https://issues.apache.org/jira/browse/HADOOP-8865 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.0.0 >Reporter: Jianbin Wei > Fix For: 3.0.0 > > Attachments: HADOOP-8865.patch, HADOOP-8865.patch, HADOOP-8865.patch > > > We need to log warn when loading deprecated properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8865) log warn when loading deprecated properties
[ https://issues.apache.org/jira/browse/HADOOP-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8865: Status: Patch Available (was: Open) > log warn when loading deprecated properties > --- > > Key: HADOOP-8865 > URL: https://issues.apache.org/jira/browse/HADOOP-8865 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.0.0 >Reporter: Jianbin Wei > Fix For: 3.0.0 > > Attachments: HADOOP-8865.patch, HADOOP-8865.patch > > > We need to log warn when loading deprecated properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8865) log warn when loading deprecated properties
[ https://issues.apache.org/jira/browse/HADOOP-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8865: Status: Open (was: Patch Available) > log warn when loading deprecated properties > --- > > Key: HADOOP-8865 > URL: https://issues.apache.org/jira/browse/HADOOP-8865 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.0.0 >Reporter: Jianbin Wei > Fix For: 3.0.0 > > Attachments: HADOOP-8865.patch, HADOOP-8865.patch > > > We need to log warn when loading deprecated properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8865) log warn when loading deprecated properties
[ https://issues.apache.org/jira/browse/HADOOP-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8865: Attachment: HADOOP-8865.patch > log warn when loading deprecated properties > --- > > Key: HADOOP-8865 > URL: https://issues.apache.org/jira/browse/HADOOP-8865 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.0.0 >Reporter: Jianbin Wei > Fix For: 3.0.0 > > Attachments: HADOOP-8865.patch, HADOOP-8865.patch > > > We need to log warn when loading deprecated properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8776) Provide an option in test-patch that can enable / disable compiling native code
[
https://issues.apache.org/jira/browse/HADOOP-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466122#comment-13466122
]
Jianbin Wei commented on HADOOP-8776:
-
I agree that it is needed. But I would agree more with
{quote}
default it to true on Linux, and false on MacOS
{quote}
We can put in HADOOP-8744 that this needs to be enabled again by default.
It is much more convenient and error-proof.
> Provide an option in test-patch that can enable / disable compiling native
> code
> ---
>
> Key: HADOOP-8776
> URL: https://issues.apache.org/jira/browse/HADOOP-8776
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0
>Reporter: Hemanth Yamijala
>Assignee: Hemanth Yamijala
>Priority: Minor
> Attachments: HADOOP-8776.patch, HADOOP-8776.patch, HADOOP-8776.patch
>
>
> The test-patch script in Hadoop source runs a native compile with the patch.
> On platforms like MAC, there are issues with the native compile that make it
> difficult to use test-patch. This JIRA is to try and provide an option to
> make the native compilation optional.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8865) log warn when loading deprecated properties
[ https://issues.apache.org/jira/browse/HADOOP-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8865: Attachment: HADOOP-8865.patch > log warn when loading deprecated properties > --- > > Key: HADOOP-8865 > URL: https://issues.apache.org/jira/browse/HADOOP-8865 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.0.0 >Reporter: Jianbin Wei > Fix For: 3.0.0 > > Attachments: HADOOP-8865.patch > > > We need to log warn when loading deprecated properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8865) log warn when loading deprecated properties
[ https://issues.apache.org/jira/browse/HADOOP-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8865: Status: Patch Available (was: Open) > log warn when loading deprecated properties > --- > > Key: HADOOP-8865 > URL: https://issues.apache.org/jira/browse/HADOOP-8865 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.0.0 >Reporter: Jianbin Wei > Fix For: 3.0.0 > > Attachments: HADOOP-8865.patch > > > We need to log warn when loading deprecated properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8865) log warn when loading deprecated properties
[
https://issues.apache.org/jira/browse/HADOOP-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465742#comment-13465742
]
Jianbin Wei commented on HADOOP-8865:
-
Yes I saw the code and it intends to do that only when the properties is being
set but not being loaded.
Following is some log after my initial change
{{[ ../hadoop-3.0.0-SNAPSHOT] ⚡ sbin/start-dfs.sh
12/09/28 09:54:03 WARN (Configuration.java:warnOnceIfDeprecated(826))
conf.Configuration: io.bytes.per.checksum is deprecated. Instead, use
dfs.bytes-per-checksum in core-default.xml
12/09/28 09:54:03 WARN (Configuration.java:warnOnceIfDeprecated(826))
conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS in
core-default.xml
12/09/28 09:54:03 WARN (Configuration.java:warnOnceIfDeprecated(826))
conf.Configuration: io.bytes.per.checksum is deprecated. Instead, use
dfs.bytes-per-checksum in core-default.xml
12/09/28 09:54:03 WARN (Configuration.java:warnOnceIfDeprecated(826))
conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS in
core-default.xml
12/09/28 09:54:03 WARN (Configuration.java:warnOnceIfDeprecated(826))
conf.Configuration: io.bytes.per.checksum is deprecated. Instead, use
dfs.bytes-per-checksum in core-default.xml}}
> log warn when loading deprecated properties
> ---
>
> Key: HADOOP-8865
> URL: https://issues.apache.org/jira/browse/HADOOP-8865
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.0.0
>Reporter: Jianbin Wei
> Fix For: 3.0.0
>
>
> We need to log warn when loading deprecated properties.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8865) log warn when loading deprecated properties
[ https://issues.apache.org/jira/browse/HADOOP-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465717#comment-13465717 ] Jianbin Wei commented on HADOOP-8865: - Such log warn can (1) avoid potential use of deprecated properties in code (https://issues.apache.org/jira/browse/MAPREDUCE-4692) (2) speedup the deprecation for further improvement in the future > log warn when loading deprecated properties > --- > > Key: HADOOP-8865 > URL: https://issues.apache.org/jira/browse/HADOOP-8865 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.0.0 >Reporter: Jianbin Wei > Fix For: 3.0.0 > > > We need to log warn when loading deprecated properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8865) log warn when loading deprecated properties
Jianbin Wei created HADOOP-8865: --- Summary: log warn when loading deprecated properties Key: HADOOP-8865 URL: https://issues.apache.org/jira/browse/HADOOP-8865 Project: Hadoop Common Issue Type: Improvement Reporter: Jianbin Wei We need to log warn when loading deprecated properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8865) log warn when loading deprecated properties
[ https://issues.apache.org/jira/browse/HADOOP-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8865: Component/s: conf Affects Version/s: 3.0.0 Fix Version/s: 3.0.0 > log warn when loading deprecated properties > --- > > Key: HADOOP-8865 > URL: https://issues.apache.org/jira/browse/HADOOP-8865 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.0.0 >Reporter: Jianbin Wei > Fix For: 3.0.0 > > > We need to log warn when loading deprecated properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8862) remove deprecated properties used in default configurations
[ https://issues.apache.org/jira/browse/HADOOP-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8862: Status: Patch Available (was: Open) This is XML change and no unit test is needed. > remove deprecated properties used in default configurations > --- > > Key: HADOOP-8862 > URL: https://issues.apache.org/jira/browse/HADOOP-8862 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.0.0 >Reporter: Jianbin Wei > Fix For: 3.0.0 > > Attachments: HADOOP-8862.patch > > > We need to remove the deprecated properties included in the default > configurations, such as core-default.xml and core-site.xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8862) remove deprecated properties used in default configurations
[ https://issues.apache.org/jira/browse/HADOOP-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8862: Attachment: HADOOP-8862.patch > remove deprecated properties used in default configurations > --- > > Key: HADOOP-8862 > URL: https://issues.apache.org/jira/browse/HADOOP-8862 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.0.0 >Reporter: Jianbin Wei > Fix For: 3.0.0 > > Attachments: HADOOP-8862.patch > > > We need to remove the deprecated properties included in the default > configurations, such as core-default.xml and core-site.xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8862) remove deprecated properties used in default configurations
[ https://issues.apache.org/jira/browse/HADOOP-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464899#comment-13464899 ] Jianbin Wei commented on HADOOP-8862: - So far I found following deprecated configuration file -- fs.default.namecore-default.xml fs.default.namecore-site.xml io.bytes.per.checksum core-default.xml > remove deprecated properties used in default configurations > --- > > Key: HADOOP-8862 > URL: https://issues.apache.org/jira/browse/HADOOP-8862 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.0.0 >Reporter: Jianbin Wei > Fix For: 3.0.0 > > > We need to remove the deprecated properties included in the default > configurations, such as core-default.xml and core-site.xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8862) remove deprecated properties used in default configurations
Jianbin Wei created HADOOP-8862: --- Summary: remove deprecated properties used in default configurations Key: HADOOP-8862 URL: https://issues.apache.org/jira/browse/HADOOP-8862 Project: Hadoop Common Issue Type: Improvement Components: conf Affects Versions: 3.0.0 Reporter: Jianbin Wei Fix For: 3.0.0 We need to remove the deprecated properties included in the default configurations, such as core-default.xml and core-site.xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8717) JAVA_HOME detected in hadoop-config.sh under OS X does not work
[ https://issues.apache.org/jira/browse/HADOOP-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8717: Attachment: HADOOP-8717.patch > JAVA_HOME detected in hadoop-config.sh under OS X does not work > --- > > Key: HADOOP-8717 > URL: https://issues.apache.org/jira/browse/HADOOP-8717 > Project: Hadoop Common > Issue Type: Bug > Components: conf > Environment: OS: Darwin 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr > 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64 > java version "1.6.0_33" > Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720) > Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode) >Reporter: Jianbin Wei >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-8717.patch, HADOOP-8717.patch, HADOOP-8717.patch, > HADOOP-8717.patch > > > After setting up a single node hadoop on mac, copy some text file to it and > run > $ hadoop jar > ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar > wordcount /file.txt output > It reports > 12/08/21 15:32:18 INFO Job.java:mapreduce.Job:1265: Running job: > job_1345588312126_0001 > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1286: Job > job_1345588312126_0001 running in uber mode : false > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1293: map 0% reduce 0% > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1306: Job > job_1345588312126_0001 failed with state FAILED due to: Application > application_1345588312126_0001 failed 1 times due to AM Container for > appattempt_1345588312126_0001_01 exited with exitCode: 127 due to: > .Failing this attempt.. Failing the application. > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1311: Counters: 0 > $ cat > /tmp/logs/application_1345588312126_0001/container_1345588312126_0001_01_01/stderr > /bin/bash: /bin/java: No such file or directory > The detected JAVA_HOME is not used somehow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8717) JAVA_HOME detected in hadoop-config.sh under OS X does not work
[ https://issues.apache.org/jira/browse/HADOOP-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8717: Status: Patch Available (was: Open) > JAVA_HOME detected in hadoop-config.sh under OS X does not work > --- > > Key: HADOOP-8717 > URL: https://issues.apache.org/jira/browse/HADOOP-8717 > Project: Hadoop Common > Issue Type: Bug > Components: conf > Environment: OS: Darwin 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr > 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64 > java version "1.6.0_33" > Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720) > Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode) >Reporter: Jianbin Wei >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-8717.patch, HADOOP-8717.patch, HADOOP-8717.patch, > HADOOP-8717.patch > > > After setting up a single node hadoop on mac, copy some text file to it and > run > $ hadoop jar > ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar > wordcount /file.txt output > It reports > 12/08/21 15:32:18 INFO Job.java:mapreduce.Job:1265: Running job: > job_1345588312126_0001 > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1286: Job > job_1345588312126_0001 running in uber mode : false > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1293: map 0% reduce 0% > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1306: Job > job_1345588312126_0001 failed with state FAILED due to: Application > application_1345588312126_0001 failed 1 times due to AM Container for > appattempt_1345588312126_0001_01 exited with exitCode: 127 due to: > .Failing this attempt.. Failing the application. > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1311: Counters: 0 > $ cat > /tmp/logs/application_1345588312126_0001/container_1345588312126_0001_01_01/stderr > /bin/bash: /bin/java: No such file or directory > The detected JAVA_HOME is not used somehow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8717) JAVA_HOME detected in hadoop-config.sh under OS X does not work
[ https://issues.apache.org/jira/browse/HADOOP-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8717: Status: Open (was: Patch Available) > JAVA_HOME detected in hadoop-config.sh under OS X does not work > --- > > Key: HADOOP-8717 > URL: https://issues.apache.org/jira/browse/HADOOP-8717 > Project: Hadoop Common > Issue Type: Bug > Components: conf > Environment: OS: Darwin 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr > 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64 > java version "1.6.0_33" > Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720) > Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode) >Reporter: Jianbin Wei >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-8717.patch, HADOOP-8717.patch, HADOOP-8717.patch > > > After setting up a single node hadoop on mac, copy some text file to it and > run > $ hadoop jar > ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar > wordcount /file.txt output > It reports > 12/08/21 15:32:18 INFO Job.java:mapreduce.Job:1265: Running job: > job_1345588312126_0001 > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1286: Job > job_1345588312126_0001 running in uber mode : false > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1293: map 0% reduce 0% > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1306: Job > job_1345588312126_0001 failed with state FAILED due to: Application > application_1345588312126_0001 failed 1 times due to AM Container for > appattempt_1345588312126_0001_01 exited with exitCode: 127 due to: > .Failing this attempt.. Failing the application. > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1311: Counters: 0 > $ cat > /tmp/logs/application_1345588312126_0001/container_1345588312126_0001_01_01/stderr > /bin/bash: /bin/java: No such file or directory > The detected JAVA_HOME is not used somehow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8717) JAVA_HOME detected in hadoop-config.sh under OS X does not work
[ https://issues.apache.org/jira/browse/HADOOP-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8717: Attachment: HADOOP-8717.patch > JAVA_HOME detected in hadoop-config.sh under OS X does not work > --- > > Key: HADOOP-8717 > URL: https://issues.apache.org/jira/browse/HADOOP-8717 > Project: Hadoop Common > Issue Type: Bug > Components: conf > Environment: OS: Darwin 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr > 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64 > java version "1.6.0_33" > Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720) > Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode) >Reporter: Jianbin Wei >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-8717.patch, HADOOP-8717.patch, HADOOP-8717.patch > > > After setting up a single node hadoop on mac, copy some text file to it and > run > $ hadoop jar > ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar > wordcount /file.txt output > It reports > 12/08/21 15:32:18 INFO Job.java:mapreduce.Job:1265: Running job: > job_1345588312126_0001 > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1286: Job > job_1345588312126_0001 running in uber mode : false > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1293: map 0% reduce 0% > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1306: Job > job_1345588312126_0001 failed with state FAILED due to: Application > application_1345588312126_0001 failed 1 times due to AM Container for > appattempt_1345588312126_0001_01 exited with exitCode: 127 due to: > .Failing this attempt.. Failing the application. > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1311: Counters: 0 > $ cat > /tmp/logs/application_1345588312126_0001/container_1345588312126_0001_01_01/stderr > /bin/bash: /bin/java: No such file or directory > The detected JAVA_HOME is not used somehow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6851) Fix '$bin' path duplication in setup scripts
[ https://issues.apache.org/jira/browse/HADOOP-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443248#comment-13443248 ] Jianbin Wei commented on HADOOP-6851: - I am not sure. But I guess you can specify the version you patch against when submitting the patch. I found it is not compatible with trunk and just notice that your patch is for 0.22. > Fix '$bin' path duplication in setup scripts > > > Key: HADOOP-6851 > URL: https://issues.apache.org/jira/browse/HADOOP-6851 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Reporter: Nicolas Spiegelberg >Priority: Trivial > Attachments: HADOOP-6851-0.22.patch > > > I have my bash environment setup to echo absolute pathnames when a relative > one is specified in 'cd'. This caused problems with all the Hadoop bash > scripts because the script accidentally sets the $bin variable twice in this > setup. (e.g. would set $bin="/path/bin/hadoop\n/path/bin/hadoop"). > This jira is for common scripts. I filed a separate jira for HDFS scripts, > which share the same pattern. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6851) Fix '$bin' path duplication in setup scripts
[ https://issues.apache.org/jira/browse/HADOOP-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442960#comment-13442960 ] Jianbin Wei commented on HADOOP-6851: - Hi Lane, you may want to reattach a new patch and click the "submit patch" to kick off the automatic hadoop QA process. In my machine, I got patch apply failure. ⚡ dev-support/test-patch.sh ~/Downloads/HADOOP-6851-0.22.patch Running in developer mode == == Testing patch for HADOOP-6851-0.22.patch. == == Patch file /Users/jianbwei/Downloads/HADOOP-6851-0.22.patch copied to /tmp The patch does not appear to apply with p0 to p2 PATCH APPLICATION FAILED -1 overall. -1 patch. The patch command could not apply the patch. == == Finished build. == == > Fix '$bin' path duplication in setup scripts > > > Key: HADOOP-6851 > URL: https://issues.apache.org/jira/browse/HADOOP-6851 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Reporter: Nicolas Spiegelberg >Priority: Trivial > Attachments: HADOOP-6851-0.22.patch > > > I have my bash environment setup to echo absolute pathnames when a relative > one is specified in 'cd'. This caused problems with all the Hadoop bash > scripts because the script accidentally sets the $bin variable twice in this > setup. (e.g. would set $bin="/path/bin/hadoop\n/path/bin/hadoop"). > This jira is for common scripts. I filed a separate jira for HDFS scripts, > which share the same pattern. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8717) JAVA_HOME detected in hadoop-config.sh under OS X does not work
[ https://issues.apache.org/jira/browse/HADOOP-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8717: Component/s: (was: bin) conf > JAVA_HOME detected in hadoop-config.sh under OS X does not work > --- > > Key: HADOOP-8717 > URL: https://issues.apache.org/jira/browse/HADOOP-8717 > Project: Hadoop Common > Issue Type: Bug > Components: conf > Environment: OS: Darwin 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr > 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64 > java version "1.6.0_33" > Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720) > Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode) >Reporter: Jianbin Wei >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-8717.patch, HADOOP-8717.patch > > > After setting up a single node hadoop on mac, copy some text file to it and > run > $ hadoop jar > ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar > wordcount /file.txt output > It reports > 12/08/21 15:32:18 INFO Job.java:mapreduce.Job:1265: Running job: > job_1345588312126_0001 > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1286: Job > job_1345588312126_0001 running in uber mode : false > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1293: map 0% reduce 0% > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1306: Job > job_1345588312126_0001 failed with state FAILED due to: Application > application_1345588312126_0001 failed 1 times due to AM Container for > appattempt_1345588312126_0001_01 exited with exitCode: 127 due to: > .Failing this attempt.. Failing the application. > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1311: Counters: 0 > $ cat > /tmp/logs/application_1345588312126_0001/container_1345588312126_0001_01_01/stderr > /bin/bash: /bin/java: No such file or directory > The detected JAVA_HOME is not used somehow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8717) JAVA_HOME detected in hadoop-config.sh under OS X does not work
[ https://issues.apache.org/jira/browse/HADOOP-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442750#comment-13442750 ] Jianbin Wei commented on HADOOP-8717: - After the patch, I start dfs and yarn and rerun the job successfully. ⚡ bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar wordcount /file.txt output3 12/08/27 14:52:34 INFO input.FileInputFormat: Total input paths to process : 1 12/08/27 14:52:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 12/08/27 14:52:34 WARN snappy.LoadSnappy: Snappy native library not loaded 12/08/27 14:52:34 INFO mapreduce.JobSubmitter: number of splits:1 12/08/27 14:52:34 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar 12/08/27 14:52:34 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 12/08/27 14:52:34 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class 12/08/27 14:52:34 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 12/08/27 14:52:34 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name 12/08/27 14:52:34 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class 12/08/27 14:52:34 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 12/08/27 14:52:34 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 12/08/27 14:52:34 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 12/08/27 14:52:34 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 12/08/27 14:52:34 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 12/08/27 14:52:35 INFO mapred.ResourceMgrDelegate: Submitted application application_1346104327473_0002 to ResourceManager at /0.0.0.0:8032 12/08/27 14:52:35 INFO mapreduce.Job: The url to track the job: http://LM-SJN-00714134:8088/proxy/application_1346104327473_0002/ 12/08/27 14:52:35 INFO mapreduce.Job: Running job: job_1346104327473_0002 12/08/27 14:52:42 INFO mapreduce.Job: Job job_1346104327473_0002 running in uber mode : false 12/08/27 14:52:42 INFO mapreduce.Job: map 0% reduce 0% 12/08/27 14:52:46 INFO mapreduce.Job: map 100% reduce 0% 12/08/27 14:52:50 INFO mapreduce.Job: map 100% reduce 100% 12/08/27 14:52:50 INFO mapreduce.Job: Job job_1346104327473_0002 completed successfully 12/08/27 14:52:50 INFO mapreduce.Job: Counters: 43 File System Counters FILE: Number of bytes read=3267 FILE: Number of bytes written=131819 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=2767 HDFS: Number of bytes written=2279 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Rack-local map tasks=1 Total time spent by all maps in occupied slots (ms)=15752 Total time spent by all reduces in occupied slots (ms)=18328 Map-Reduce Framework Map input records=87 Map output records=317 Map output bytes=3881 Map output materialized bytes=3027 Input split bytes=95 Combine input records=317 Combine output records=186 Reduce input groups=186 Reduce shuffle bytes=3027 Reduce input records=186 Reduce output records=186 Spilled Records=372 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=15 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=282660864 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=2672 File Output Format Counters Bytes Written=2279 > JAVA_HOME detected in hadoop-config.sh under OS X does not work > ---
[jira] [Updated] (HADOOP-8717) JAVA_HOME detected in hadoop-config.sh under OS X does not work
[ https://issues.apache.org/jira/browse/HADOOP-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8717: Attachment: HADOOP-8717.patch > JAVA_HOME detected in hadoop-config.sh under OS X does not work > --- > > Key: HADOOP-8717 > URL: https://issues.apache.org/jira/browse/HADOOP-8717 > Project: Hadoop Common > Issue Type: Bug > Components: bin > Environment: OS: Darwin 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr > 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64 > java version "1.6.0_33" > Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720) > Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode) >Reporter: Jianbin Wei >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-8717.patch, HADOOP-8717.patch > > > After setting up a single node hadoop on mac, copy some text file to it and > run > $ hadoop jar > ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar > wordcount /file.txt output > It reports > 12/08/21 15:32:18 INFO Job.java:mapreduce.Job:1265: Running job: > job_1345588312126_0001 > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1286: Job > job_1345588312126_0001 running in uber mode : false > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1293: map 0% reduce 0% > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1306: Job > job_1345588312126_0001 failed with state FAILED due to: Application > application_1345588312126_0001 failed 1 times due to AM Container for > appattempt_1345588312126_0001_01 exited with exitCode: 127 due to: > .Failing this attempt.. Failing the application. > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1311: Counters: 0 > $ cat > /tmp/logs/application_1345588312126_0001/container_1345588312126_0001_01_01/stderr > /bin/bash: /bin/java: No such file or directory > The detected JAVA_HOME is not used somehow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8717) JAVA_HOME detected in hadoop-config.sh under OS X does not work
[ https://issues.apache.org/jira/browse/HADOOP-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8717: Status: Patch Available (was: Open) > JAVA_HOME detected in hadoop-config.sh under OS X does not work > --- > > Key: HADOOP-8717 > URL: https://issues.apache.org/jira/browse/HADOOP-8717 > Project: Hadoop Common > Issue Type: Bug > Components: bin > Environment: OS: Darwin 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr > 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64 > java version "1.6.0_33" > Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720) > Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode) >Reporter: Jianbin Wei >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-8717.patch, HADOOP-8717.patch > > > After setting up a single node hadoop on mac, copy some text file to it and > run > $ hadoop jar > ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar > wordcount /file.txt output > It reports > 12/08/21 15:32:18 INFO Job.java:mapreduce.Job:1265: Running job: > job_1345588312126_0001 > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1286: Job > job_1345588312126_0001 running in uber mode : false > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1293: map 0% reduce 0% > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1306: Job > job_1345588312126_0001 failed with state FAILED due to: Application > application_1345588312126_0001 failed 1 times due to AM Container for > appattempt_1345588312126_0001_01 exited with exitCode: 127 due to: > .Failing this attempt.. Failing the application. > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1311: Counters: 0 > $ cat > /tmp/logs/application_1345588312126_0001/container_1345588312126_0001_01_01/stderr > /bin/bash: /bin/java: No such file or directory > The detected JAVA_HOME is not used somehow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8717) JAVA_HOME detected in hadoop-config.sh under OS X does not work
[ https://issues.apache.org/jira/browse/HADOOP-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8717: Affects Version/s: (was: 3.0.0) Status: Open (was: Patch Available) > JAVA_HOME detected in hadoop-config.sh under OS X does not work > --- > > Key: HADOOP-8717 > URL: https://issues.apache.org/jira/browse/HADOOP-8717 > Project: Hadoop Common > Issue Type: Bug > Components: bin > Environment: OS: Darwin 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr > 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64 > java version "1.6.0_33" > Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720) > Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode) >Reporter: Jianbin Wei >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-8717.patch > > > After setting up a single node hadoop on mac, copy some text file to it and > run > $ hadoop jar > ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar > wordcount /file.txt output > It reports > 12/08/21 15:32:18 INFO Job.java:mapreduce.Job:1265: Running job: > job_1345588312126_0001 > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1286: Job > job_1345588312126_0001 running in uber mode : false > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1293: map 0% reduce 0% > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1306: Job > job_1345588312126_0001 failed with state FAILED due to: Application > application_1345588312126_0001 failed 1 times due to AM Container for > appattempt_1345588312126_0001_01 exited with exitCode: 127 due to: > .Failing this attempt.. Failing the application. > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1311: Counters: 0 > $ cat > /tmp/logs/application_1345588312126_0001/container_1345588312126_0001_01_01/stderr > /bin/bash: /bin/java: No such file or directory > The detected JAVA_HOME is not used somehow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8719) workaround Hadoop logs errors upon startup on OS X 10.7
[
https://issues.apache.org/jira/browse/HADOOP-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianbin Wei updated HADOOP-8719:
Status: Patch Available (was: Open)
resubmit the patch with renamed 'macos' to 'MAC_OSX'. Thanks Harsh.
> workaround Hadoop logs errors upon startup on OS X 10.7
> ---
>
> Key: HADOOP-8719
> URL: https://issues.apache.org/jira/browse/HADOOP-8719
> Project: Hadoop Common
> Issue Type: Bug
> Environment: Mac OS X 10.7, Java 1.6.0_26
>Reporter: Jianbin Wei
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-8719.patch, HADOOP-8719.patch, HADOOP-8719.patch,
> HADOOP-8719.patch
>
>
> When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs
> the following errors:
> 2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
> SCDynamicStore
> Hadoop does seem to function properly despite this.
> The workaround takes only 10 minutes.
> There are numerous discussions about this:
> google "Unable to load realm mapping info from SCDynamicStore" returns 1770
> hits. Each one has many discussions.
> Assume each discussion take only 5 minute, a 10-minute fix can save ~150
> hours. This does not count much search of this issue and its
> solution/workaround, which can easily hit (wasted) thousands of hours!!!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8719) workaround Hadoop logs errors upon startup on OS X 10.7
[
https://issues.apache.org/jira/browse/HADOOP-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianbin Wei updated HADOOP-8719:
Status: Open (was: Patch Available)
> workaround Hadoop logs errors upon startup on OS X 10.7
> ---
>
> Key: HADOOP-8719
> URL: https://issues.apache.org/jira/browse/HADOOP-8719
> Project: Hadoop Common
> Issue Type: Bug
> Environment: Mac OS X 10.7, Java 1.6.0_26
>Reporter: Jianbin Wei
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-8719.patch, HADOOP-8719.patch, HADOOP-8719.patch,
> HADOOP-8719.patch
>
>
> When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs
> the following errors:
> 2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
> SCDynamicStore
> Hadoop does seem to function properly despite this.
> The workaround takes only 10 minutes.
> There are numerous discussions about this:
> google "Unable to load realm mapping info from SCDynamicStore" returns 1770
> hits. Each one has many discussions.
> Assume each discussion take only 5 minute, a 10-minute fix can save ~150
> hours. This does not count much search of this issue and its
> solution/workaround, which can easily hit (wasted) thousands of hours!!!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8719) workaround Hadoop logs errors upon startup on OS X 10.7
[
https://issues.apache.org/jira/browse/HADOOP-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianbin Wei updated HADOOP-8719:
Attachment: HADOOP-8719.patch
rename 'macos' to 'MAC_OSX'
> workaround Hadoop logs errors upon startup on OS X 10.7
> ---
>
> Key: HADOOP-8719
> URL: https://issues.apache.org/jira/browse/HADOOP-8719
> Project: Hadoop Common
> Issue Type: Bug
> Environment: Mac OS X 10.7, Java 1.6.0_26
>Reporter: Jianbin Wei
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-8719.patch, HADOOP-8719.patch, HADOOP-8719.patch,
> HADOOP-8719.patch
>
>
> When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs
> the following errors:
> 2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
> SCDynamicStore
> Hadoop does seem to function properly despite this.
> The workaround takes only 10 minutes.
> There are numerous discussions about this:
> google "Unable to load realm mapping info from SCDynamicStore" returns 1770
> hits. Each one has many discussions.
> Assume each discussion take only 5 minute, a 10-minute fix can save ~150
> hours. This does not count much search of this issue and its
> solution/workaround, which can easily hit (wasted) thousands of hours!!!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8719) workaround Hadoop logs errors upon startup on OS X 10.7
[
https://issues.apache.org/jira/browse/HADOOP-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442673#comment-13442673
]
Jianbin Wei commented on HADOOP-8719:
-
Used the style from
hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh, the check
for cygwin. The one used over there should also be changed for consistency I
think.
> workaround Hadoop logs errors upon startup on OS X 10.7
> ---
>
> Key: HADOOP-8719
> URL: https://issues.apache.org/jira/browse/HADOOP-8719
> Project: Hadoop Common
> Issue Type: Bug
> Environment: Mac OS X 10.7, Java 1.6.0_26
>Reporter: Jianbin Wei
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-8719.patch, HADOOP-8719.patch, HADOOP-8719.patch,
> HADOOP-8719.patch
>
>
> When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs
> the following errors:
> 2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
> SCDynamicStore
> Hadoop does seem to function properly despite this.
> The workaround takes only 10 minutes.
> There are numerous discussions about this:
> google "Unable to load realm mapping info from SCDynamicStore" returns 1770
> hits. Each one has many discussions.
> Assume each discussion take only 5 minute, a 10-minute fix can save ~150
> hours. This does not count much search of this issue and its
> solution/workaround, which can easily hit (wasted) thousands of hours!!!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8717) JAVA_HOME detected in hadoop-config.sh under OS X does not work
[ https://issues.apache.org/jira/browse/HADOOP-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8717: Fix Version/s: 3.0.0 Status: Patch Available (was: Open) used non-array to set the JAVA_HOME under mac > JAVA_HOME detected in hadoop-config.sh under OS X does not work > --- > > Key: HADOOP-8717 > URL: https://issues.apache.org/jira/browse/HADOOP-8717 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 3.0.0 > Environment: OS: Darwin 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr > 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64 > java version "1.6.0_33" > Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720) > Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode) >Reporter: Jianbin Wei >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-8717.patch > > > After setting up a single node hadoop on mac, copy some text file to it and > run > $ hadoop jar > ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar > wordcount /file.txt output > It reports > 12/08/21 15:32:18 INFO Job.java:mapreduce.Job:1265: Running job: > job_1345588312126_0001 > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1286: Job > job_1345588312126_0001 running in uber mode : false > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1293: map 0% reduce 0% > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1306: Job > job_1345588312126_0001 failed with state FAILED due to: Application > application_1345588312126_0001 failed 1 times due to AM Container for > appattempt_1345588312126_0001_01 exited with exitCode: 127 due to: > .Failing this attempt.. Failing the application. > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1311: Counters: 0 > $ cat > /tmp/logs/application_1345588312126_0001/container_1345588312126_0001_01_01/stderr > /bin/bash: /bin/java: No such file or directory > The detected JAVA_HOME is not used somehow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8717) JAVA_HOME detected in hadoop-config.sh under OS X does not work
[ https://issues.apache.org/jira/browse/HADOOP-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianbin Wei updated HADOOP-8717: Attachment: HADOOP-8717.patch > JAVA_HOME detected in hadoop-config.sh under OS X does not work > --- > > Key: HADOOP-8717 > URL: https://issues.apache.org/jira/browse/HADOOP-8717 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 3.0.0 > Environment: OS: Darwin 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr > 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64 > java version "1.6.0_33" > Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720) > Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode) >Reporter: Jianbin Wei >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-8717.patch > > > After setting up a single node hadoop on mac, copy some text file to it and > run > $ hadoop jar > ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar > wordcount /file.txt output > It reports > 12/08/21 15:32:18 INFO Job.java:mapreduce.Job:1265: Running job: > job_1345588312126_0001 > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1286: Job > job_1345588312126_0001 running in uber mode : false > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1293: map 0% reduce 0% > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1306: Job > job_1345588312126_0001 failed with state FAILED due to: Application > application_1345588312126_0001 failed 1 times due to AM Container for > appattempt_1345588312126_0001_01 exited with exitCode: 127 due to: > .Failing this attempt.. Failing the application. > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1311: Counters: 0 > $ cat > /tmp/logs/application_1345588312126_0001/container_1345588312126_0001_01_01/stderr > /bin/bash: /bin/java: No such file or directory > The detected JAVA_HOME is not used somehow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8719) workaround Hadoop logs errors upon startup on OS X 10.7
[
https://issues.apache.org/jira/browse/HADOOP-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442604#comment-13442604
]
Jianbin Wei commented on HADOOP-8719:
-
This changes the options used by hadoop-env.sh and yarn-env.sh. After the
patch, I did manual test and verified that sbin/[start|stop]-[dfs|yarn].sh do
not show the annoying messages any more.
> workaround Hadoop logs errors upon startup on OS X 10.7
> ---
>
> Key: HADOOP-8719
> URL: https://issues.apache.org/jira/browse/HADOOP-8719
> Project: Hadoop Common
> Issue Type: Bug
> Environment: Mac OS X 10.7, Java 1.6.0_26
>Reporter: Jianbin Wei
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-8719.patch, HADOOP-8719.patch, HADOOP-8719.patch
>
>
> When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs
> the following errors:
> 2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
> SCDynamicStore
> Hadoop does seem to function properly despite this.
> The workaround takes only 10 minutes.
> There are numerous discussions about this:
> google "Unable to load realm mapping info from SCDynamicStore" returns 1770
> hits. Each one has many discussions.
> Assume each discussion take only 5 minute, a 10-minute fix can save ~150
> hours. This does not count much search of this issue and its
> solution/workaround, which can easily hit (wasted) thousands of hours!!!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8719) workaround Hadoop logs errors upon startup on OS X 10.7
[
https://issues.apache.org/jira/browse/HADOOP-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianbin Wei updated HADOOP-8719:
Status: Open (was: Patch Available)
> workaround Hadoop logs errors upon startup on OS X 10.7
> ---
>
> Key: HADOOP-8719
> URL: https://issues.apache.org/jira/browse/HADOOP-8719
> Project: Hadoop Common
> Issue Type: Bug
> Environment: Mac OS X 10.7, Java 1.6.0_26
>Reporter: Jianbin Wei
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-8719.patch, HADOOP-8719.patch, HADOOP-8719.patch
>
>
> When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs
> the following errors:
> 2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
> SCDynamicStore
> Hadoop does seem to function properly despite this.
> The workaround takes only 10 minutes.
> There are numerous discussions about this:
> google "Unable to load realm mapping info from SCDynamicStore" returns 1770
> hits. Each one has many discussions.
> Assume each discussion take only 5 minute, a 10-minute fix can save ~150
> hours. This does not count much search of this issue and its
> solution/workaround, which can easily hit (wasted) thousands of hours!!!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8719) workaround Hadoop logs errors upon startup on OS X 10.7
[
https://issues.apache.org/jira/browse/HADOOP-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianbin Wei updated HADOOP-8719:
Status: Patch Available (was: Open)
> workaround Hadoop logs errors upon startup on OS X 10.7
> ---
>
> Key: HADOOP-8719
> URL: https://issues.apache.org/jira/browse/HADOOP-8719
> Project: Hadoop Common
> Issue Type: Bug
> Environment: Mac OS X 10.7, Java 1.6.0_26
>Reporter: Jianbin Wei
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-8719.patch, HADOOP-8719.patch, HADOOP-8719.patch
>
>
> When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs
> the following errors:
> 2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
> SCDynamicStore
> Hadoop does seem to function properly despite this.
> The workaround takes only 10 minutes.
> There are numerous discussions about this:
> google "Unable to load realm mapping info from SCDynamicStore" returns 1770
> hits. Each one has many discussions.
> Assume each discussion take only 5 minute, a 10-minute fix can save ~150
> hours. This does not count much search of this issue and its
> solution/workaround, which can easily hit (wasted) thousands of hours!!!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8719) workaround Hadoop logs errors upon startup on OS X 10.7
[
https://issues.apache.org/jira/browse/HADOOP-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianbin Wei updated HADOOP-8719:
Attachment: HADOOP-8719.patch
> workaround Hadoop logs errors upon startup on OS X 10.7
> ---
>
> Key: HADOOP-8719
> URL: https://issues.apache.org/jira/browse/HADOOP-8719
> Project: Hadoop Common
> Issue Type: Bug
> Environment: Mac OS X 10.7, Java 1.6.0_26
>Reporter: Jianbin Wei
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-8719.patch, HADOOP-8719.patch, HADOOP-8719.patch
>
>
> When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs
> the following errors:
> 2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
> SCDynamicStore
> Hadoop does seem to function properly despite this.
> The workaround takes only 10 minutes.
> There are numerous discussions about this:
> google "Unable to load realm mapping info from SCDynamicStore" returns 1770
> hits. Each one has many discussions.
> Assume each discussion take only 5 minute, a 10-minute fix can save ~150
> hours. This does not count much search of this issue and its
> solution/workaround, which can easily hit (wasted) thousands of hours!!!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8719) workaround Hadoop logs errors upon startup on OS X 10.7
[
https://issues.apache.org/jira/browse/HADOOP-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianbin Wei updated HADOOP-8719:
Description:
When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs the
following errors:
2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
SCDynamicStore
Hadoop does seem to function properly despite this.
The workaround takes only 10 minutes.
There are numerous discussions about this:
google "Unable to load realm mapping info from SCDynamicStore" returns 1770
hits. Each one has many discussions.
Assume each discussion take only 5 minute, a 10-minute fix can save ~150 hours.
This does not count much search of this issue and its solution/workaround,
which can easily hit (wasted) thousands of hours!!!
was:
When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs the
following errors:
2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
SCDynamicStore
Hadoop does seem to function properly despite this.
There are numerous discussions about this:
google "Unable to load realm mapping info from SCDynamicStore" returns 1770
hits. Each one has many discussions.
Assume each discussion take only 5 minute, a 10-minute fix can save ~150 hours.
This does not count much search of this issue and its solution/workaround,
which can easily hit (wasted) thousands of hours!!!
> workaround Hadoop logs errors upon startup on OS X 10.7
> ---
>
> Key: HADOOP-8719
> URL: https://issues.apache.org/jira/browse/HADOOP-8719
> Project: Hadoop Common
> Issue Type: Bug
> Environment: Mac OS X 10.7, Java 1.6.0_26
>Reporter: Jianbin Wei
>Priority: Minor
> Attachments: HADOOP-8719.patch
>
>
> When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs
> the following errors:
> 2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
> SCDynamicStore
> Hadoop does seem to function properly despite this.
> The workaround takes only 10 minutes.
> There are numerous discussions about this:
> google "Unable to load realm mapping info from SCDynamicStore" returns 1770
> hits. Each one has many discussions.
> Assume each discussion take only 5 minute, a 10-minute fix can save ~150
> hours. This does not count much search of this issue and its
> solution/workaround, which can easily hit (wasted) thousands of hours!!!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8719) workaround Hadoop logs errors upon startup on OS X 10.7
[
https://issues.apache.org/jira/browse/HADOOP-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianbin Wei updated HADOOP-8719:
Attachment: HADOOP-8719.patch
> workaround Hadoop logs errors upon startup on OS X 10.7
> ---
>
> Key: HADOOP-8719
> URL: https://issues.apache.org/jira/browse/HADOOP-8719
> Project: Hadoop Common
> Issue Type: Bug
> Environment: Mac OS X 10.7, Java 1.6.0_26
>Reporter: Jianbin Wei
>Priority: Minor
> Attachments: HADOOP-8719.patch
>
>
> When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs
> the following errors:
> 2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
> SCDynamicStore
> Hadoop does seem to function properly despite this.
> There are numerous discussions about this:
> google "Unable to load realm mapping info from SCDynamicStore" returns 1770
> hits. Each one has many discussions.
> Assume each discussion take only 5 minute, a 10-minute fix can save ~150
> hours. This does not count much search of this issue and its
> solution/workaround, which can easily hit (wasted) thousands of hours!!!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8719) workaround Hadoop logs errors upon startup on OS X 10.7
[
https://issues.apache.org/jira/browse/HADOOP-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianbin Wei updated HADOOP-8719:
Description:
When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs the
following errors:
2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
SCDynamicStore
Hadoop does seem to function properly despite this.
There are numerous discussions about this:
google "Unable to load realm mapping info from SCDynamicStore" returns 1770
hits. Each one has many discussions.
Assume each discussion take only 5 minute, a 10-minute fix can save ~150 hours.
This does not count much search of this issue and its solution/workaround,
which can easily hit (wasted) thousands of hours!!!
was:
When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs the
following errors:
2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
SCDynamicStore
Hadoop does seem to function properly despite this.
> workaround Hadoop logs errors upon startup on OS X 10.7
> ---
>
> Key: HADOOP-8719
> URL: https://issues.apache.org/jira/browse/HADOOP-8719
> Project: Hadoop Common
> Issue Type: Bug
> Environment: Mac OS X 10.7, Java 1.6.0_26
>Reporter: Jianbin Wei
>Priority: Minor
>
> When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs
> the following errors:
> 2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
> SCDynamicStore
> Hadoop does seem to function properly despite this.
> There are numerous discussions about this:
> google "Unable to load realm mapping info from SCDynamicStore" returns 1770
> hits. Each one has many discussions.
> Assume each discussion take only 5 minute, a 10-minute fix can save ~150
> hours. This does not count much search of this issue and its
> solution/workaround, which can easily hit (wasted) thousands of hours!!!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8719) workaround Hadoop logs errors upon startup on OS X 10.7
Jianbin Wei created HADOOP-8719:
---
Summary: workaround Hadoop logs errors upon startup on OS X 10.7
Key: HADOOP-8719
URL: https://issues.apache.org/jira/browse/HADOOP-8719
Project: Hadoop Common
Issue Type: Bug
Environment: Mac OS X 10.7, Java 1.6.0_26
Reporter: Jianbin Wei
Priority: Minor
When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs the
following errors:
2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
SCDynamicStore
Hadoop does seem to function properly despite this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8717) JAVA_HOME detected in hadoop-config.sh under OS X does not work
[ https://issues.apache.org/jira/browse/HADOOP-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439182#comment-13439182 ] Jianbin Wei commented on HADOOP-8717: - If use non-array assignment of JAVA_HOME it works. diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh index aa971f9..02e5f15 100644 --- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh +++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh @@ -136,9 +136,9 @@ if [[ -z $JAVA_HOME ]]; then # On OSX use java_home (or /Library for older versions) if [ "Darwin" == "$(uname -s)" ]; then if [ -x /usr/libexec/java_home ]; then - export JAVA_HOME=($(/usr/libexec/java_home)) + export JAVA_HOME=$(/usr/libexec/java_home) else - export JAVA_HOME=(/Library/Java/Home) + export JAVA_HOME=/Library/Java/Home fi fi > JAVA_HOME detected in hadoop-config.sh under OS X does not work > --- > > Key: HADOOP-8717 > URL: https://issues.apache.org/jira/browse/HADOOP-8717 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 3.0.0 > Environment: OS: Darwin 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr > 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64 > java version "1.6.0_33" > Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720) > Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode) >Reporter: Jianbin Wei >Priority: Minor > > After setting up a single node hadoop on mac, copy some text file to it and > run > $ hadoop jar > ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar > wordcount /file.txt output > It reports > 12/08/21 15:32:18 INFO Job.java:mapreduce.Job:1265: Running job: > job_1345588312126_0001 > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1286: Job > job_1345588312126_0001 running in uber mode : false > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1293: map 0% reduce 0% > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1306: Job > job_1345588312126_0001 failed with state FAILED due to: Application > application_1345588312126_0001 failed 1 times due to AM Container for > appattempt_1345588312126_0001_01 exited with exitCode: 127 due to: > .Failing this attempt.. Failing the application. > 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1311: Counters: 0 > $ cat > /tmp/logs/application_1345588312126_0001/container_1345588312126_0001_01_01/stderr > /bin/bash: /bin/java: No such file or directory > The detected JAVA_HOME is not used somehow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8717) JAVA_HOME detected in hadoop-config.sh under OS X does not work
Jianbin Wei created HADOOP-8717: --- Summary: JAVA_HOME detected in hadoop-config.sh under OS X does not work Key: HADOOP-8717 URL: https://issues.apache.org/jira/browse/HADOOP-8717 Project: Hadoop Common Issue Type: Bug Components: bin Affects Versions: 3.0.0 Environment: OS: Darwin 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64 java version "1.6.0_33" Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720) Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode) Reporter: Jianbin Wei Priority: Minor After setting up a single node hadoop on mac, copy some text file to it and run $ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar wordcount /file.txt output It reports 12/08/21 15:32:18 INFO Job.java:mapreduce.Job:1265: Running job: job_1345588312126_0001 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1286: Job job_1345588312126_0001 running in uber mode : false 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1293: map 0% reduce 0% 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1306: Job job_1345588312126_0001 failed with state FAILED due to: Application application_1345588312126_0001 failed 1 times due to AM Container for appattempt_1345588312126_0001_01 exited with exitCode: 127 due to: .Failing this attempt.. Failing the application. 12/08/21 15:32:22 INFO Job.java:mapreduce.Job:1311: Counters: 0 $ cat /tmp/logs/application_1345588312126_0001/container_1345588312126_0001_01_01/stderr /bin/bash: /bin/java: No such file or directory The detected JAVA_HOME is not used somehow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
