[jira] [Updated] (HADOOP-14498) HADOOP_OPTIONAL_TOOLS not parsed correctly
[
https://issues.apache.org/jira/browse/HADOOP-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-14498:
Target Version/s: 3.5.0 (was: 3.4.0)
> HADOOP_OPTIONAL_TOOLS not parsed correctly
> --
>
> Key: HADOOP-14498
> URL: https://issues.apache.org/jira/browse/HADOOP-14498
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha1
>Reporter: Mingliang Liu
>Assignee: Sean Mackrory
>Priority: Major
> Attachments: HADOOP-14498.001.patch, HADOOP-14498.002.patch,
> HADOOP-14498.003.patch
>
>
> # This will make hadoop-azure not show up in the hadoop classpath, though
> both hadoop-aws and hadoop-azure-datalake are in the
> classpath.{code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake"
> {code}
> # And if we put only hadoop-azure and hadoop-aws, both of them are shown in
> the classpath.
> {code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws"
> {code}
> This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we
> make some assumptions that hadoop tool modules have a single "-" in names,
> and the _hadoop-azure-datalake_ overrides the _hadoop-azure_. Or any other
> assumptions about the {{${project.artifactId\}}}?
> Ping [~aw].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14498) HADOOP_OPTIONAL_TOOLS not parsed correctly
[
https://issues.apache.org/jira/browse/HADOOP-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HADOOP-14498:
--
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> HADOOP_OPTIONAL_TOOLS not parsed correctly
> --
>
> Key: HADOOP-14498
> URL: https://issues.apache.org/jira/browse/HADOOP-14498
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha1
>Reporter: Mingliang Liu
>Assignee: Sean Mackrory
>Priority: Major
> Attachments: HADOOP-14498.001.patch, HADOOP-14498.002.patch,
> HADOOP-14498.003.patch
>
>
> # This will make hadoop-azure not show up in the hadoop classpath, though
> both hadoop-aws and hadoop-azure-datalake are in the
> classpath.{code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake"
> {code}
> # And if we put only hadoop-azure and hadoop-aws, both of them are shown in
> the classpath.
> {code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws"
> {code}
> This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we
> make some assumptions that hadoop tool modules have a single "-" in names,
> and the _hadoop-azure-datalake_ overrides the _hadoop-azure_. Or any other
> assumptions about the {{${project.artifactId\}}}?
> Ping [~aw].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14498) HADOOP_OPTIONAL_TOOLS not parsed correctly
[
https://issues.apache.org/jira/browse/HADOOP-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-14498:
-
Priority: Major (was: Critical)
Description:
# This will make hadoop-azure not show up in the hadoop classpath, though both
hadoop-aws and hadoop-azure-datalake are in the
classpath.{code:title=hadoop-env.sh}
export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake"
{code}
# And if we put only hadoop-azure and hadoop-aws, both of them are shown in the
classpath.
{code:title=hadoop-env.sh}
export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws"
{code}
This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we make
some assumptions that hadoop tool modules have a single "-" in names, and the
_hadoop-azure-datalake_ overrides the _hadoop-azure_. Or any other assumptions
about the {{${project.artifactId\}}}?
Ping [~aw].
was:
# This will make hadoop-azure not show up in the hadoop classpath, though both
hadoop-aws and hadoop-azure-datalake are in the
classpath.{code:title=hadoop-env.sh}
export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake"
{code}
# And if we put only hadoop-azure and hadoop-aws, both of them are shown in the
classpath.
{code:title=hadoop-env.sh}
export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws"
{code}
This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we make
some assumptions that hadoop tool modules have a single "-" in names, and the
_hadoop-azure-datalake_ overrides the _hadoop-azure_. Or any other assumptions
about the {{${project.artifactId\}}}?
Ping [~aw].
I'm dropping this in priority after offline discussion from Sean.
> HADOOP_OPTIONAL_TOOLS not parsed correctly
> --
>
> Key: HADOOP-14498
> URL: https://issues.apache.org/jira/browse/HADOOP-14498
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha1
>Reporter: Mingliang Liu
>Assignee: Sean Mackrory
> Attachments: HADOOP-14498.001.patch, HADOOP-14498.002.patch,
> HADOOP-14498.003.patch
>
>
> # This will make hadoop-azure not show up in the hadoop classpath, though
> both hadoop-aws and hadoop-azure-datalake are in the
> classpath.{code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake"
> {code}
> # And if we put only hadoop-azure and hadoop-aws, both of them are shown in
> the classpath.
> {code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws"
> {code}
> This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we
> make some assumptions that hadoop tool modules have a single "-" in names,
> and the _hadoop-azure-datalake_ overrides the _hadoop-azure_. Or any other
> assumptions about the {{${project.artifactId\}}}?
> Ping [~aw].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14498) HADOOP_OPTIONAL_TOOLS not parsed correctly
[
https://issues.apache.org/jira/browse/HADOOP-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Mackrory updated HADOOP-14498:
---
Attachment: HADOOP-14498.003.patch
.003 patch address shellcheck issues and refactors the loop into a shared
hadoop_join_array function.
> HADOOP_OPTIONAL_TOOLS not parsed correctly
> --
>
> Key: HADOOP-14498
> URL: https://issues.apache.org/jira/browse/HADOOP-14498
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha1
>Reporter: Mingliang Liu
>Assignee: Sean Mackrory
>Priority: Critical
> Attachments: HADOOP-14498.001.patch, HADOOP-14498.002.patch,
> HADOOP-14498.003.patch
>
>
> # This will make hadoop-azure not show up in the hadoop classpath, though
> both hadoop-aws and hadoop-azure-datalake are in the
> classpath.{code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake"
> {code}
> # And if we put only hadoop-azure and hadoop-aws, both of them are shown in
> the classpath.
> {code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws"
> {code}
> This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we
> make some assumptions that hadoop tool modules have a single "-" in names,
> and the _hadoop-azure-datalake_ overrides the _hadoop-azure_. Or any other
> assumptions about the {{${project.artifactId\}}}?
> Ping [~aw].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14498) HADOOP_OPTIONAL_TOOLS not parsed correctly
[
https://issues.apache.org/jira/browse/HADOOP-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Mackrory updated HADOOP-14498:
---
Attachment: HADOOP-14498.002.patch
Attaching said patch It'd be nice to replace that inline loop with a join
function. I don't immediately see one - do we have a file for
non-Hadoop-related helper functions like that? Or should I just throw it in
hadoop-functions.sh? (logging off for the night - will iterate tomorrow).
> HADOOP_OPTIONAL_TOOLS not parsed correctly
> --
>
> Key: HADOOP-14498
> URL: https://issues.apache.org/jira/browse/HADOOP-14498
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha1
>Reporter: Mingliang Liu
>Assignee: Sean Mackrory
>Priority: Critical
> Attachments: HADOOP-14498.001.patch, HADOOP-14498.002.patch
>
>
> # This will make hadoop-azure not show up in the hadoop classpath, though
> both hadoop-aws and hadoop-azure-datalake are in the
> classpath.{code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake"
> {code}
> # And if we put only hadoop-azure and hadoop-aws, both of them are shown in
> the classpath.
> {code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws"
> {code}
> This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we
> make some assumptions that hadoop tool modules have a single "-" in names,
> and the _hadoop-azure-datalake_ overrides the _hadoop-azure_. Or any other
> assumptions about the {{${project.artifactId\}}}?
> Ping [~aw].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14498) HADOOP_OPTIONAL_TOOLS not parsed correctly
[
https://issues.apache.org/jira/browse/HADOOP-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Mackrory updated HADOOP-14498:
---
Status: Patch Available (was: Open)
> HADOOP_OPTIONAL_TOOLS not parsed correctly
> --
>
> Key: HADOOP-14498
> URL: https://issues.apache.org/jira/browse/HADOOP-14498
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha1
>Reporter: Mingliang Liu
>Assignee: Sean Mackrory
>Priority: Critical
> Attachments: HADOOP-14498.001.patch
>
>
> # This will make hadoop-azure not show up in the hadoop classpath, though
> both hadoop-aws and hadoop-azure-datalake are in the
> classpath.{code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake"
> {code}
> # And if we put only hadoop-azure and hadoop-aws, both of them are shown in
> the classpath.
> {code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws"
> {code}
> This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we
> make some assumptions that hadoop tool modules have a single "-" in names,
> and the _hadoop-azure-datalake_ overrides the _hadoop-azure_. Or any other
> assumptions about the {{${project.artifactId\}}}?
> Ping [~aw].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14498) HADOOP_OPTIONAL_TOOLS not parsed correctly
[
https://issues.apache.org/jira/browse/HADOOP-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Mackrory updated HADOOP-14498:
---
Attachment: HADOOP-14498.001.patch
Now that HADOOP-13595 is in I had a look at this. It is indeed the regex
matching logic in hadoop_add_param being overzealous. That function does
already state an assumption that it is space-delimited, so I used spaces or
line boundaries to ensure a full-word patch instead of just a partial one.
Rather than have it use arrays, let's fix this function for all the other
places it's used instead of using something from HADOOP-13595 instead.
Attaching a patch. I got a clean Yetus run locally, but it wasn't running bats
tests for some reason. I ran the hadoop_add_param tests manually but not the
others, and some unrelated Java test failures seem to be blocking it (unless
there's a way to bypass that and easily run all bats tests?)
> HADOOP_OPTIONAL_TOOLS not parsed correctly
> --
>
> Key: HADOOP-14498
> URL: https://issues.apache.org/jira/browse/HADOOP-14498
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha1
>Reporter: Mingliang Liu
>Assignee: Sean Mackrory
>Priority: Critical
> Attachments: HADOOP-14498.001.patch
>
>
> # This will make hadoop-azure not show up in the hadoop classpath, though
> both hadoop-aws and hadoop-azure-datalake are in the
> classpath.{code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake"
> {code}
> # And if we put only hadoop-azure and hadoop-aws, both of them are shown in
> the classpath.
> {code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws"
> {code}
> This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we
> make some assumptions that hadoop tool modules have a single "-" in names,
> and the _hadoop-azure-datalake_ overrides the _hadoop-azure_. Or any other
> assumptions about the {{${project.artifactId\}}}?
> Ping [~aw].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14498) HADOOP_OPTIONAL_TOOLS not parsed correctly
[
https://issues.apache.org/jira/browse/HADOOP-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14498:
---
Priority: Critical (was: Major)
> HADOOP_OPTIONAL_TOOLS not parsed correctly
> --
>
> Key: HADOOP-14498
> URL: https://issues.apache.org/jira/browse/HADOOP-14498
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha1
>Reporter: Mingliang Liu
>Priority: Critical
>
> # This will make hadoop-azure not show up in the hadoop classpath, though
> both hadoop-aws and hadoop-azure-datalake are in the
> classpath.{code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake"
> {code}
> # And if we put only hadoop-azure and hadoop-aws, both of them are shown in
> the classpath.
> {code:title=hadoop-env.sh}
> export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws"
> {code}
> This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we
> make some assumptions that hadoop tool modules have a single "-" in names,
> and the _hadoop-azure-datalake_ overrides the _hadoop-azure_. Or any other
> assumptions about the {{${project.artifactId\}}}?
> Ping [~aw].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
