[
https://issues.apache.org/jira/browse/HADOOP-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18103676#comment-18103676
]
ASF GitHub Bot commented on HADOOP-15019:
-----------------------------------------
deepujain opened a new pull request, #8670:
URL: https://github.com/apache/hadoop/pull/8670
### Description of PR
**Summary**
When `HADOOP_USER_CLASSPATH_FIRST=true`, a user-supplied classpath entry that
is already present in Hadoop's computed classpath can stay in its older
position instead of moving to the front. This happens because duplicate
classpath entries were treated as a no-op even when they were being re-added
with `before`.
This patch keeps the existing de-duplication behavior, but when a duplicate
entry is re-added with `before`, it is moved to the front of `CLASSPATH`.
That preserves the expected override behavior for cases such as
`HADOOP_CONF_DIR` also being present in `HADOOP_CLASSPATH`.
**Change**
- `hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh`:
update `hadoop_add_classpath()` so duplicate entries are reordered to the
front when added with `before`.
-
`hadoop-common-project/hadoop-common/src/test/scripts/hadoop_add_classpath.bats`:
add a regression that proves a duplicate entry can be moved to the front.
-
`hadoop-common-project/hadoop-common/src/test/scripts/hadoop_finalize_classpath.bats`:
add a higher-level regression that matches the JIRA scenario where
`HADOOP_CONF_DIR` is duplicated in `HADOOP_CLASSPATH` while
`HADOOP_USER_CLASSPATH_FIRST=true`.
**Evidence it works**
- `bash -n
hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh`
- `git diff --check`
- Direct shell validation of duplicate reordering:
`/bin/bash -lc 'set -eo pipefail; export HADOOP_SHELL_SCRIPT_DEBUG=true;
export QATESTMODE=true; export HADOOP_LIBEXEC_DIR=$(cd -P
hadoop-common-project/hadoop-common/src/main/bin >/dev/null && pwd -P); source
hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh;
tmpdir=$(mktemp -d); mkdir -p "$tmpdir/first" "$tmpdir/second"; CLASSPATH="";
hadoop_add_classpath "$tmpdir/first" after; hadoop_add_classpath
"$tmpdir/second" after; hadoop_add_classpath "$tmpdir/second" before; test
"$CLASSPATH" = "$tmpdir/second:$tmpdir/first"; rm -rf "$tmpdir"'`
- Direct shell validation of the JIRA-style finalize flow:
`/bin/bash -lc 'set -eo pipefail; export HADOOP_SHELL_SCRIPT_DEBUG=true;
export QATESTMODE=true; export HADOOP_LIBEXEC_DIR=$(cd -P
hadoop-common-project/hadoop-common/src/main/bin >/dev/null && pwd -P); source
hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh;
tmpdir=$(mktemp -d); mkdir -p "$tmpdir/new" "$tmpdir/old"; CLASSPATH="";
HADOOP_CONF_DIR="$tmpdir"; HADOOP_CLASSPATH="$tmpdir:$tmpdir/new:$tmpdir/old";
HADOOP_USER_CLASSPATH_FIRST=true; HADOOP_USE_CLIENT_CLASSLOADER="";
hadoop_translate_cygwin_path() { true; }; hadoop_finalize_classpath; test
"$CLASSPATH" = "$tmpdir:$tmpdir/new:$tmpdir/old"; rm -rf "$tmpdir"'`
- `bats` is not installed in this local environment, so the new `.bats`
regressions were validated through the equivalent sourced-function shell
invocations above.
**JIRA**
Fixes HADOOP-15019
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint
declared according to the connector-specific documentation? Not
applicable to this shell-script change.
- [x] If adding new dependencies to the code, are these dependencies licensed
in a way that is compatible for inclusion under
[ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
No new dependencies are added.
- [x] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files? No license or notice changes are required.
### AI Tooling
Contains content generated by Codex.
If an AI tool was used:
- [x] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
> Hadoop shell script classpath de-duping ignores HADOOP_USER_CLASSPATH_FIRST
> ----------------------------------------------------------------------------
>
> Key: HADOOP-15019
> URL: https://issues.apache.org/jira/browse/HADOOP-15019
> Project: Hadoop Common
> Issue Type: Bug
> Components: bin
> Reporter: Philip Martin
> Priority: Major
>
> If a user sets {{HADOOP_USER_CLASSPATH_FIRST=true}} and furthermore includes
> a directory that's already in Hadoop's classpath via {{HADOOP_CLASSPATH}},
> that directory will appear later than it should in the eventual $CLASSPATH. I
> believe this is because the de-duping at
> https://github.com/apache/hadoop/blob/cbc632d9abf08c56a7fc02be51b2718af30bad28/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L1200
> is ignoring the "before/after" parameter.
> My way of reproduction, first build the following trivial Java program:
> {code}
> $cat Test.java
> public class Test {
> public static void main(String[]args) {
> System.out.println(System.getenv().get("CLASSPATH"));
> }
> }
> $javac Test.java
> $jar cf test.jar Test.class
> {code}
> With that, if you happen to have an entry in HADOOP_CLASSPATH that matches
> what Hadoop would produce, you'll find the ordering not honored. It's easiest
> to reproduce this with a match for HADOOP_CONF_DIR, as in the second case
> below:
> {code}
> # As you'd expect, /usr/share is first!
> $HADOOP_CONF_DIR=/etc HADOOP_USER_CLASSPATH_FIRST="true"
> HADOOP_CLASSPATH=/usr/share:/tmp:/bin bin/hadoop jar test.jar Test | tr ':'
> '\n' | grep -n . | grep '/usr/share'
> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
> 1:/usr/share
> # Surprise! /usr/share is now in the 3rd line, even thought it was first in
> HADOOP_CLASSPATH.
> $HADOOP_CONF_DIR=/usr/share HADOOP_USER_CLASSPATH_FIRST="true"
> HADOOP_CLASSPATH=/usr/share:/tmp:/bin bin/hadoop jar test.jar Test | tr ':'
> '\n' | grep -n . | grep '/usr/share'
> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
> 3:/usr/share
> {code}
> To re-iterate, what's surprising is that you can make an entry that's first
> in HADOOP_USER_CLASSPATH show up not first in the resulting classpath.
> I ran into this configuring {{bin/hive}} with a confdir that was being used
> for both HDFS and Hive, and flailing as to why my {{log4j2.properties}}
> wasn't being read. The one in my conf dir was lower in my classpath than one
> bundled in some Hive jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
