[
https://issues.apache.org/jira/browse/HADOOP-16649?focusedWorklogId=503559&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-503559
]
ASF GitHub Bot logged work on HADOOP-16649:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 22/Oct/20 05:57
Start Date: 22/Oct/20 05:57
Worklog Time Spent: 10m
Work Description: aw-was-here commented on pull request #2385:
URL: https://github.com/apache/hadoop/pull/2385#issuecomment-714249153
Just to save everyone a lot of time and suffering:
This approach will break a lot of things in very unexpected ways (doing a
search on everywhere hadoop_add_params is called should make this clear).
hadoop_add_param was specifically built for partial matches because the
HADOOP_OPTS command line can't really do exact matches and this was a quick way
to prevent duplicate options. The unit test failure in
hadoop_finalize_hadoop_heap was intended to provide a hint that "yar they be
dragons here." I should have written better tests, but given it took like 2
years just to get most of this code in over the total @#$@#$ that was in hadoop
2.x ...
When I wrote the code originally, we didn't have a need for exact matches
anywhere (HADOOP_OPTIONAL_TOOLS wasn't written yet). It was written and
committed to 3.x. Then the HADOOP_OPTIONAL_TOOLS code was written but that
would be the only place where an exact match would be useful and we didn't have
any sooo... I just re-used hadoop_add_param with the (clearly faulty)
assumption that people would test their code on Hadoop 3.x. But since the
azure team didn't bother to test with hadoop 3.x until it was too late... At
this point, I was getting tired of the Hadoop politics and bailed, leaving this
furball hanging around.
Anyway, the *real* fix for this is to create a new function that converts
HADOOP_OPTIONAL_TOOLS to an array and then do an exact match, looping over the
array. I think there is code to do that now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 503559)
Time Spent: 1h 10m (was: 1h)
> Defining hadoop-azure and hadoop-azure-datalake in HADOOP_OPTIONAL_TOOLS will
> ignore hadoop-azure
> -------------------------------------------------------------------------------------------------
>
> Key: HADOOP-16649
> URL: https://issues.apache.org/jira/browse/HADOOP-16649
> Project: Hadoop Common
> Issue Type: Bug
> Components: bin
> Affects Versions: 3.2.1
> Environment: Shell, but it also trickles down into all code using
> `FileSystem`
> Reporter: Tom Lous
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When defining both `hadoop-azure` and `hadoop-azure-datalake` in
> HADOOP_OPTIONAL_TOOLS in `conf/hadoop-env.sh`, `hadoop-azure` will get
> ignored.
> eg setting this:
> HADOOP_OPTIONAL_TOOLS="hadoop-azure-datalake,hadoop-azure"
>
> with debug on:
>
> DEBUG: Profiles: importing
> /opt/hadoop/libexec/shellprofile.d/hadoop-azure-datalake.sh
> DEBUG: HADOOP_SHELL_PROFILES accepted hadoop-azure-datalake
> DEBUG: Profiles: importing /opt/hadoop/libexec/shellprofile.d/hadoop-azure.sh
> DEBUG: HADOOP_SHELL_PROFILES
> DEBUG: HADOOP_SHELL_PROFILES declined hadoop-azure hadoop-azure
>
> whereas:
>
> HADOOP_OPTIONAL_TOOLS="hadoop-azure"
>
> with debug on:
> DEBUG: Profiles: importing /opt/hadoop/libexec/shellprofile.d/hadoop-azure.sh
> DEBUG: HADOOP_SHELL_PROFILES accepted hadoop-azure
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
