[jira] [Commented] (BEAM-7937) Support Hadoop 3.x on Hadoop File System
[ https://issues.apache.org/jira/browse/BEAM-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122716#comment-17122716 ] Beam JIRA Bot commented on BEAM-7937: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Support Hadoop 3.x on Hadoop File System > > > Key: BEAM-7937 > URL: https://issues.apache.org/jira/browse/BEAM-7937 > Project: Beam > Issue Type: Sub-task > Components: io-java-hadoop-file-system >Reporter: Reenu Saluja >Priority: P2 > Labels: stale-P2 > > I try to get a Beam pipeline to have input from Azure Data Lake Storage Gen > 2. ADLS Gen 2 Support Hadoop 3.2+. I tried using Apache Beam 2.8.1 later on > 2.14.0. I am getting below error: > Error: Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No > FileSystem for scheme "wasbs" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7937) Support Hadoop 3.x on Hadoop File System
[ https://issues.apache.org/jira/browse/BEAM-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910490#comment-16910490 ] Reenu Saluja commented on BEAM-7937: To add more details about the problem statement, We are already using hadoop-azure 2.7 version. with this API we are able to download data from ADLS Gen 2 on a shared storage. Than with Beam api transformation of data is happening. But we are looking for api through with Beam code can directly connect with ADLS Gen 2. For e.g. to read data from kafka, there is option for KafkaIO read(). p.apply(KafkaIO.read() Do we have similar function for Hadoop also(ADLS Gen2 ) ? > Support Hadoop 3.x on Hadoop File System > > > Key: BEAM-7937 > URL: https://issues.apache.org/jira/browse/BEAM-7937 > Project: Beam > Issue Type: Improvement > Components: io-java-hadoop-file-system >Reporter: Reenu Saluja >Priority: Major > > I try to get a Beam pipeline to have input from Azure Data Lake Storage Gen > 2. ADLS Gen 2 Support Hadoop 3.2+. I tried using Apache Beam 2.8.1 later on > 2.14.0. I am getting below error: > Error: Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No > FileSystem for scheme "wasbs" -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-7937) Support Hadoop 3.x on Hadoop File System
[ https://issues.apache.org/jira/browse/BEAM-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905116#comment-16905116 ] Ismaël Mejía commented on BEAM-7937: The requirement of Hadoop 3.2+ is only to support ADLS Gen 2 via `hadoop-azure-datalake`, since you are trying use `wasbs`. This should work with earlier version of Hadoop (including 2.7.x) by suing instead 'hadoop-azure'. https://hadoop.apache.org/docs/current/hadoop-azure/index.html > Support Hadoop 3.x on Hadoop File System > > > Key: BEAM-7937 > URL: https://issues.apache.org/jira/browse/BEAM-7937 > Project: Beam > Issue Type: Improvement > Components: io-java-hadoop-file-system >Reporter: Reenu Saluja >Priority: Major > > I try to get a Beam pipeline to have input from Azure Data Lake Storage Gen > 2. ADLS Gen 2 Support Hadoop 3.2+. I tried using Apache Beam 2.8.1 later on > 2.14.0. I am getting below error: > Error: Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No > FileSystem for scheme "wasbs" -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (BEAM-7937) Support Hadoop 3.x on Hadoop File System
[ https://issues.apache.org/jira/browse/BEAM-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905112#comment-16905112 ] Ismaël Mejía commented on BEAM-7937: Thanks for reporting this, we do not include yet tests or official support for Hadoop 3.x on Beam, but probably it would be achievable via provided dependencies. I just tested and there is a compilation bug that we need to address first. > Support Hadoop 3.x on Hadoop File System > > > Key: BEAM-7937 > URL: https://issues.apache.org/jira/browse/BEAM-7937 > Project: Beam > Issue Type: Improvement > Components: io-java-hadoop-file-system >Reporter: Reenu Saluja >Priority: Major > > I try to get a Beam pipeline to have input from Azure Data Lake Storage Gen > 2. ADLS Gen 2 Support Hadoop 3.2+. I tried using Apache Beam 2.8.1 later on > 2.14.0. I am getting below error: > Error: Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No > FileSystem for scheme "wasbs" -- This message was sent by Atlassian JIRA (v7.6.14#76016)
