[jira] [Commented] (HADOOP-15544) ABFS: validate packing, transient classpath, hadoop fs CLI
[
https://issues.apache.org/jira/browse/HADOOP-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627831#comment-16627831
]
Steve Loughran commented on HADOOP-15544:
-
you shouldn't need to be editing HADOOP_CLASSPATH; the .hadooprc file should do
that for you
> ABFS: validate packing, transient classpath, hadoop fs CLI
> --
>
> Key: HADOOP-15544
> URL: https://issues.apache.org/jira/browse/HADOOP-15544
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: dependencies.txt
>
>
> Validate the packaging and dependencies of ABFS
> * hadoop-cloud-storage artifact to export everything needed
> * {{hadoop fs -ls abfs://path}} to work in ASF distributions
> * check transient CP (e.g spark)
> Spark master;s hadoop-cloud module depends on hadoop-cloud-storage if you
> build with the hadoop-3.1 profile, so it should automatically get in there.
> Just need to check that it picks it up too
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15544) ABFS: validate packing, transient classpath, hadoop fs CLI
[
https://issues.apache.org/jira/browse/HADOOP-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554894#comment-16554894
]
Da Zhou commented on HADOOP-15544:
--
Thank you [~ste...@apache.org] for sharing the steps. At my end I can run the
hadoop cmd without issue.
What I did: I followed the steps you shared, then append *$HADOOP_CLASSPATH*
with the required jars path (azure jar path and
/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/share/hadoop/tools/lib/*).
After that I tried WASB:
{code:java}
./bin/hadoop fs -ls
wasb://TEST_CONTAINER_NAME@TEST_ACCOUNT.blob.core.windows.net/{code}
Then tried ABFS:
{code:java}
./bin/hadoop fs -ls
abfs://TEST_CONTAINER_NAME@TEST_ACCOUNT.dfs.core.windows.net/{code}
Both can return the results successfully.
Could you share your setting for core-site.xml and the failure message with
me? I guess it might be related to the configuration.
Thanks,
Da
> ABFS: validate packing, transient classpath, hadoop fs CLI
> --
>
> Key: HADOOP-15544
> URL: https://issues.apache.org/jira/browse/HADOOP-15544
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: dependencies.txt
>
>
> Validate the packaging and dependencies of ABFS
> * hadoop-cloud-storage artifact to export everything needed
> * {{hadoop fs -ls abfs://path}} to work in ASF distributions
> * check transient CP (e.g spark)
> Spark master;s hadoop-cloud module depends on hadoop-cloud-storage if you
> build with the hadoop-3.1 profile, so it should automatically get in there.
> Just need to check that it picks it up too
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15544) ABFS: validate packing, transient classpath, hadoop fs CLI
[
https://issues.apache.org/jira/browse/HADOOP-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553664#comment-16553664
]
Steve Loughran commented on HADOOP-15544:
-
Also worth exploring is my cloudstore/storediag module, which I keep to debug
this stuff: [https://github.com/steveloughran/cloudstore/releases]
{code}
bin/hadoop -jar cloudstore-2.8.jar abfs://storeuri/path
{code}
That attempts to load the abfs, look for (and log) classes/dependencies it
knows of, then does some basic IO. It's intended to be for remote debugging of
support calls, so can be expanded to do stuff about hostname and proxy lookup,
once the details are known
> ABFS: validate packing, transient classpath, hadoop fs CLI
> --
>
> Key: HADOOP-15544
> URL: https://issues.apache.org/jira/browse/HADOOP-15544
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: dependencies.txt
>
>
> Validate the packaging and dependencies of ABFS
> * hadoop-cloud-storage artifact to export everything needed
> * {{hadoop fs -ls abfs://path}} to work in ASF distributions
> * check transient CP (e.g spark)
> Spark master;s hadoop-cloud module depends on hadoop-cloud-storage if you
> build with the hadoop-3.1 profile, so it should automatically get in there.
> Just need to check that it picks it up too
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15544) ABFS: validate packing, transient classpath, hadoop fs CLI
[
https://issues.apache.org/jira/browse/HADOOP-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553662#comment-16553662
]
Steve Loughran commented on HADOOP-15544:
-
no need for a cluster
Run: mvn package -Pdist -DskipTests -Dmaven.javadoc.skip=true -DskipShade
this puts things under hadoop-dist/target/hadoop-3.2.0-SNAPSHOT
* cd there
* copy into etc/hadoop under there your log4j and core-site.xml settings
* in your ~/.hadooprc file, add what you want for Hadoop modules to always
load; here's one of mine
{code}
> cat ~/.hadooprc
hadoop_add_to_classpath_tools hadoop-aws hadoop-azure hadoop-azuredatalake
{code}
then go {{bin/hadoop fs -ls abfs://something}} to see what happens
> ABFS: validate packing, transient classpath, hadoop fs CLI
> --
>
> Key: HADOOP-15544
> URL: https://issues.apache.org/jira/browse/HADOOP-15544
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: dependencies.txt
>
>
> Validate the packaging and dependencies of ABFS
> * hadoop-cloud-storage artifact to export everything needed
> * {{hadoop fs -ls abfs://path}} to work in ASF distributions
> * check transient CP (e.g spark)
> Spark master;s hadoop-cloud module depends on hadoop-cloud-storage if you
> build with the hadoop-3.1 profile, so it should automatically get in there.
> Just need to check that it picks it up too
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15544) ABFS: validate packing, transient classpath, hadoop fs CLI
[
https://issues.apache.org/jira/browse/HADOOP-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553419#comment-16553419
]
Da Zhou commented on HADOOP-15544:
--
Hi [~ste...@apache.org], could you share with me the steps to test hadoop CLI
from Intellij?
Or I have to deploy a cluster for hadoop fs CLI test?
> ABFS: validate packing, transient classpath, hadoop fs CLI
> --
>
> Key: HADOOP-15544
> URL: https://issues.apache.org/jira/browse/HADOOP-15544
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: dependencies.txt
>
>
> Validate the packaging and dependencies of ABFS
> * hadoop-cloud-storage artifact to export everything needed
> * {{hadoop fs -ls abfs://path}} to work in ASF distributions
> * check transient CP (e.g spark)
> Spark master;s hadoop-cloud module depends on hadoop-cloud-storage if you
> build with the hadoop-3.1 profile, so it should automatically get in there.
> Just need to check that it picks it up too
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15544) ABFS: validate packing, transient classpath, hadoop fs CLI
[
https://issues.apache.org/jira/browse/HADOOP-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552154#comment-16552154
]
Shane Mainali commented on HADOOP-15544:
Adding [~DanielZhou] for a look on this as well, I would guess that maybe the
failures are related to the other configuration changes in HADOOP-15546?
> ABFS: validate packing, transient classpath, hadoop fs CLI
> --
>
> Key: HADOOP-15544
> URL: https://issues.apache.org/jira/browse/HADOOP-15544
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: dependencies.txt
>
>
> Validate the packaging and dependencies of ABFS
> * hadoop-cloud-storage artifact to export everything needed
> * {{hadoop fs -ls abfs://path}} to work in ASF distributions
> * check transient CP (e.g spark)
> Spark master;s hadoop-cloud module depends on hadoop-cloud-storage if you
> build with the hadoop-3.1 profile, so it should automatically get in there.
> Just need to check that it picks it up too
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15544) ABFS: validate packing, transient classpath, hadoop fs CLI
[
https://issues.apache.org/jira/browse/HADOOP-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525605#comment-16525605
]
Steve Loughran commented on HADOOP-15544:
-
CP is set up for the CLI with the right env vars; namely
{code}
export
HADOOP_OPTIONAL_TOOLS="hadoop-kafka,hadoop-aws,hadoop-aliyun,hadoop-openstack,hadoop-ozone,hadoop-azure,hadoop-azure-datalake"
{code}
If I do this, adl:// works ok, but wasb and abfs both fail
> ABFS: validate packing, transient classpath, hadoop fs CLI
> --
>
> Key: HADOOP-15544
> URL: https://issues.apache.org/jira/browse/HADOOP-15544
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: dependencies.txt
>
>
> Validate the packaging and dependencies of ABFS
> * hadoop-cloud-storage artifact to export everything needed
> * {{hadoop fs -ls abfs://path}} to work in ASF distributions
> * check transient CP (e.g spark)
> Spark master;s hadoop-cloud module depends on hadoop-cloud-storage if you
> build with the hadoop-3.1 profile, so it should automatically get in there.
> Just need to check that it picks it up too
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15544) ABFS: validate packing, transient classpath, hadoop fs CLI
[
https://issues.apache.org/jira/browse/HADOOP-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513934#comment-16513934
]
Steve Loughran commented on HADOOP-15544:
-
hadoop cloud storage marks hadoop-common as provided (intentional, to keep lots
of transient cruft out); as a result there's a leaner set of imports from the
hadoop-azure module
{code}
[INFO] +- org.apache.hadoop:hadoop-azure:jar:3.2.0-SNAPSHOT:compile
[INFO] | +- org.threadly:threadly:jar:4.9.0:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-core:jar:2.9.5:compile
[INFO] | +- com.microsoft.azure:azure-storage:jar:7.0.0:compile
[INFO] | | \- com.microsoft.azure:azure-keyvault-core:jar:1.0.0:compile
[INFO] | +- org.apache.htrace:htrace-core:jar:3.1.0-incubating:compile
[INFO] | +- com.google.inject:guice:jar:4.0:compile
[INFO] | | +- javax.inject:javax.inject:jar:1:compile
[INFO] | | \- aopalliance:aopalliance:jar:1.0:compile
[INFO] | +- joda-time:joda-time:jar:2.9.9:compile
[INFO] | \- org.eclipse.jetty:jetty-util-ajax:jar:9.3.19.v20170502:compile
{code}
I'm slightly worried there's no
{{org.apache.htrace:htrace-core4:jar:4.1.0-incubating}} on that list, though we
do expect all of hadoop-common to have come in first
Need to check downstream to see what ends up on the CP
> ABFS: validate packing, transient classpath, hadoop fs CLI
> --
>
> Key: HADOOP-15544
> URL: https://issues.apache.org/jira/browse/HADOOP-15544
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: dependencies.txt
>
>
> Validate the packaging and dependencies of ABFS
> * hadoop-cloud-storage artifact to export everything needed
> * {{hadoop fs -ls abfs://path}} to work in ASF distributions
> * check transient CP (e.g spark)
> Spark master;s hadoop-cloud module depends on hadoop-cloud-storage if you
> build with the hadoop-3.1 profile, so it should automatically get in there.
> Just need to check that it picks it up too
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15544) ABFS: validate packing, transient classpath, hadoop fs CLI
[
https://issues.apache.org/jira/browse/HADOOP-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513764#comment-16513764
]
Steve Loughran commented on HADOOP-15544:
-
Compile deps of the hadoop-azure module
{code}
[INFO] +- org.threadly:threadly:jar:4.9.0:compile
[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.9.5:compile
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.5:compile
[INFO] | \- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.5:compile
[INFO] +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile
[INFO] | \- org.apache.httpcomponents:httpcore:jar:4.4.4:compile
[INFO] +- com.microsoft.azure:azure-storage:jar:7.0.0:compile
[INFO] | \- com.microsoft.azure:azure-keyvault-core:jar:1.0.0:compile
[INFO] +- org.apache.htrace:htrace-core:jar:3.1.0-incubating:compile
[INFO] +- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:compile
[INFO] +- com.google.inject:guice:jar:4.0:compile
[INFO] | +- javax.inject:javax.inject:jar:1:compile
[INFO] | \- aopalliance:aopalliance:jar:1.0:compile
[INFO] +- com.google.guava:guava:jar:11.0.2:compile
[INFO] +- joda-time:joda-time:jar:2.9.9:compile
[INFO] +- org.eclipse.jetty:jetty-util-ajax:jar:9.3.19.v20170502:compile
{code}
> ABFS: validate packing, transient classpath, hadoop fs CLI
> --
>
> Key: HADOOP-15544
> URL: https://issues.apache.org/jira/browse/HADOOP-15544
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: dependencies.txt
>
>
> Validate the packaging and dependencies of ABFS
> * hadoop-cloud-storage artifact to export everything needed
> * {{hadoop fs -ls abfs://path}} to work in ASF distributions
> * check transient CP (e.g spark)
> Spark master;s hadoop-cloud module depends on hadoop-cloud-storage if you
> build with the hadoop-3.1 profile, so it should automatically get in there.
> Just need to check that it picks it up too
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
