[jira] [Created] (HADOOP-16405) Upgrade Wildfly Openssl version to 1.0.7.Final
Vishwajeet Dusane created HADOOP-16405: -- Summary: Upgrade Wildfly Openssl version to 1.0.7.Final Key: HADOOP-16405 URL: https://issues.apache.org/jira/browse/HADOOP-16405 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Reporter: Vishwajeet Dusane Upgrade Wildfly Openssl version to 1.0.7.Final. This version has SNI support which is essential for firewall enabled clusters along with many stability related fixes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16402) AAD MSI flow is broken
Vishwajeet Dusane created HADOOP-16402: -- Summary: AAD MSI flow is broken Key: HADOOP-16402 URL: https://issues.apache.org/jira/browse/HADOOP-16402 Project: Hadoop Common Issue Type: Bug Components: fs/adl Affects Versions: 2.9.0 Reporter: Vishwajeet Dusane Assignee: Vishwajeet Dusane For AAD with MSI flow to work, ADL driver needs to initialize MsiTokenProvider class with AAD Client id and tenant id. With the current implementation, AAD MSI flow is broken. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16182) Update storage back-end with "close" flag when application is done writing to a file
Vishwajeet Dusane created HADOOP-16182: -- Summary: Update storage back-end with "close" flag when application is done writing to a file Key: HADOOP-16182 URL: https://issues.apache.org/jira/browse/HADOOP-16182 Project: Hadoop Common Issue Type: Task Components: fs/azure Affects Versions: 3.2.0 Reporter: Vishwajeet Dusane As part of Azure Data Lake Storage Gen2 notifications design, customers are interested in knowing when a client is done writing to a file so they can take certain actions like initiate a pipeline, or replicate a file, or start certain processing. To satisfy that, ABFS client should send "close" flag during Flush rest API when invoked by OutputStream::close() API. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15965) Upgrade to ADLS SDK which has major performance improvement for ingress/egress
Vishwajeet Dusane created HADOOP-15965: -- Summary: Upgrade to ADLS SDK which has major performance improvement for ingress/egress Key: HADOOP-15965 URL: https://issues.apache.org/jira/browse/HADOOP-15965 Project: Hadoop Common Issue Type: Improvement Components: fs/adl Reporter: Vishwajeet Dusane Upgrade ADLS SDK to version 2.3.2 which has major improvements # Add special handling for 404 errors when requesting tokens from MSI # Fix liststatus response parsing when filestatus object contains array in one field. # Use wildfly openssl native binding with Java. This is a workaround to [https://bugs.openjdk.java.net/browse/JDK-8046943]issue. 2X performance boost over HTTPS. Similar to HADOOP-15669 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15851) Disable wildfly logs to the console
Vishwajeet Dusane created HADOOP-15851:
--
Summary: Disable wildfly logs to the console
Key: HADOOP-15851
URL: https://issues.apache.org/jira/browse/HADOOP-15851
Project: Hadoop Common
Issue Type: Task
Components: fs/azure
Reporter: Vishwajeet Dusane
Assignee: Vishwajeet Dusane
Fix For: 3.2.0
On loading OpenSSL library successfully, Wildfly logging messages like below
{code:java}
Oct 15, 2018 6:47:24 AM org.wildfly.openssl.SSL init
INFO: WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.0g 2 Nov 2017
{code}
These messages may fiddle with existing scripts which parses logs with a
predefined schema.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15700) ABFS: Failure in OpenSSLProvider should fall back to JSSE
Vishwajeet Dusane created HADOOP-15700:
--
Summary: ABFS: Failure in OpenSSLProvider should fall back to JSSE
Key: HADOOP-15700
URL: https://issues.apache.org/jira/browse/HADOOP-15700
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Reporter: Vishwajeet Dusane
Assignee: Vishwajeet Dusane
Failure to {{OpenSSLProvider.register()}} should fall back to default JSSE
initialization. This is needed to support Java 7 in case the HADOOP-15669 is
back-ported to support Java7.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15187) Remove mock test dependency on REST call invoked from Java SDK
Vishwajeet Dusane created HADOOP-15187: -- Summary: Remove mock test dependency on REST call invoked from Java SDK Key: HADOOP-15187 URL: https://issues.apache.org/jira/browse/HADOOP-15187 Project: Hadoop Common Issue Type: Improvement Components: fs/adl Affects Versions: 3.0.0 Reporter: Vishwajeet Dusane Assignee: Vishwajeet Dusane Cleanup unit test which mocks REST calls invoked within dependency SDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15186) Allow Azure Data Lake SDK dependency version to override from the command line
Vishwajeet Dusane created HADOOP-15186: -- Summary: Allow Azure Data Lake SDK dependency version to override from the command line Key: HADOOP-15186 URL: https://issues.apache.org/jira/browse/HADOOP-15186 Project: Hadoop Common Issue Type: Improvement Components: fs/adl Affects Versions: 3.0.0 Reporter: Vishwajeet Dusane Assignee: Vishwajeet Dusane For backward/forward release of Java SDK compatibility test against Hadoop driver. Allow Azure Data Lake Java SDK dependency version to override from command line. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14730) hasAcl property always set to false, regardless of FsPermission higher bit order
Vishwajeet Dusane created HADOOP-14730:
--
Summary: hasAcl property always set to false, regardless of
FsPermission higher bit order
Key: HADOOP-14730
URL: https://issues.apache.org/jira/browse/HADOOP-14730
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0-beta1
Reporter: Vishwajeet Dusane
Assignee: Chris Douglas
2 Unit Test cases are failing [Azure-data-lake Module
|https://github.com/apache/hadoop/blob/4966a6e26e45d7dc36e0b270066ff7c87bcd00cc/hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestGetFileStatus.java#L44-L44],
caused after HDFS-6984 commit.
Issue seems to be {{hasAcl}} is hard coded to {{false}}.
{code:java}
public FileStatus(long length, boolean isdir,
int block_replication,
long blocksize, long modification_time, long access_time,
FsPermission permission, String owner, String group,
Path symlink,
Path path) {
this(length, isdir, block_replication, blocksize, modification_time,
access_time, permission, owner, group, symlink, path,
false, false, false);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14017) User friendly name for ADLS user and group
[
https://issues.apache.org/jira/browse/HADOOP-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishwajeet Dusane reopened HADOOP-14017:
Assignee: Vishwajeet Dusane (was: John Zhuge)
Based on the discussion on the same thread with [~jzhuge]. Reopening this issue
instead of creating new feature request.
ADLS to support configuration toggle between OID and UPN (User friendly name).
> User friendly name for ADLS user and group
> --
>
> Key: HADOOP-14017
> URL: https://issues.apache.org/jira/browse/HADOOP-14017
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: John Zhuge
>Assignee: Vishwajeet Dusane
>
> ADLS displays GUID whenever user or group displayed, e.g., {{ls}},
> {{getfacl}}.
> ADLS requires GUID whenever user or group input is needed, e.g., {{setfacl}},
> {{chown}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14049) Honour AclBit flag associated to file/folder permission for Azure datalake account
Vishwajeet Dusane created HADOOP-14049:
--
Summary: Honour AclBit flag associated to file/folder permission
for Azure datalake account
Key: HADOOP-14049
URL: https://issues.apache.org/jira/browse/HADOOP-14049
Project: Hadoop Common
Issue Type: New Feature
Components: fs/adl
Affects Versions: 3.0.0-alpha3
Reporter: Vishwajeet Dusane
ADLS persist AclBit information on a file/folder. Since Java SDK 2.1.4 - AclBit
value can be retrieved using {{DirectoryEntry.aclBit}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13393) [Azure Data Lake] Correction needed in fs.defaultFS support in documentation
Vishwajeet Dusane created HADOOP-13393: -- Summary: [Azure Data Lake] Correction needed in fs.defaultFS support in documentation Key: HADOOP-13393 URL: https://issues.apache.org/jira/browse/HADOOP-13393 Project: Hadoop Common Issue Type: Bug Components: fs/azure Reporter: Vishwajeet Dusane Azure data lake does not support fs.defaultFS pointing to adls storage as of today.So documentation to point fs.defaultFS configuration is confusing to customer integrating adls configuration in there storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13392) [Azure Data Lake] OAuth2 configuration should be default set to true when AdlFileSystem is configured.
Vishwajeet Dusane created HADOOP-13392: -- Summary: [Azure Data Lake] OAuth2 configuration should be default set to true when AdlFileSystem is configured. Key: HADOOP-13392 URL: https://issues.apache.org/jira/browse/HADOOP-13392 Project: Hadoop Common Issue Type: Bug Components: fs/azure Reporter: Vishwajeet Dusane OAuth2 configuration is optional in case of WebHdfsFileSystem configuration. However OAuth2 configuration should be default set to true when AdlFileSystem is configured. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-12877) [Azure Data Lake] Add Telemetry instrumentation
Vishwajeet Dusane created HADOOP-12877: -- Summary: [Azure Data Lake] Add Telemetry instrumentation Key: HADOOP-12877 URL: https://issues.apache.org/jira/browse/HADOOP-12877 Project: Hadoop Common Issue Type: New Feature Components: fs, fs/azure, tools Reporter: Vishwajeet Dusane Assignee: Vishwajeet Dusane -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12876) [Azure Data Lake] Support for process level FileStatus cache to optimize GetFileStatus frequent opeations
Vishwajeet Dusane created HADOOP-12876: -- Summary: [Azure Data Lake] Support for process level FileStatus cache to optimize GetFileStatus frequent opeations Key: HADOOP-12876 URL: https://issues.apache.org/jira/browse/HADOOP-12876 Project: Hadoop Common Issue Type: New Feature Components: fs, fs/azure, tools Reporter: Vishwajeet Dusane Assignee: Vishwajeet Dusane Add support to cache GetFileStatus and ListStatus response locally for limited period of time. Local cache for limited period of time would optimize number of calls for GetFileStatus operation. One of the example where local limited period cache would be useful - terasort ListStatus on input directory follows with GetFileStatus operation on each file within directory. For 2048 input files in a directory would save 2048 GetFileStatus calls during start up (Using the ListStatus response to cache FileStatus instances). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
Vishwajeet Dusane created HADOOP-12875: -- Summary: [Azure Data Lake] Support for contract test and unit test cases Key: HADOOP-12875 URL: https://issues.apache.org/jira/browse/HADOOP-12875 Project: Hadoop Common Issue Type: Test Components: fs, fs/azure, tools Reporter: Vishwajeet Dusane Assignee: Vishwajeet Dusane This JIRA describes contract test and unit test cases support for azure data lake file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12666) Support Windows Azure Data Lake - as a file system in Hadoop
vishwajeet dusane created HADOOP-12666:
--
Summary: Support Windows Azure Data Lake - as a file system in
Hadoop
Key: HADOOP-12666
URL: https://issues.apache.org/jira/browse/HADOOP-12666
Project: Hadoop Common
Issue Type: New Feature
Components: tools
Reporter: vishwajeet dusane
h2. Description
This JIRA describes a new file system implementation for accessing Windows
Azure Data Lake Store (ADL) from within Hadoop. This would enable existing
Hadoop applications such has MR, HIVE, Hbase etc.., to use ADL store as input
or output.
ADL is ultra-high capacity, Optimized for massive throughput with rich
management and security features. More details available at
https://azure.microsoft.com/en-us/services/data-lake-store/
h2. High level design
ADL file system exposes RESTful interfaces compatible with WebHdfs
specification 2.7.1.
At a high level, the code here extends the SWebHdfsFileSystem class to provide
an implementation for accessing ADL storage; the scheme ADL is used for
accessing it over HTTPS. We use the URI scheme:
{code}adl:///path/to/file{code}
to address individual Files/Folders. Tests are implemented mostly using a
Contract implementation for the ADL functionality, with an option to test
against a real ADL storage if configured.
h2. Credits and history
This has been ongoing work for a while, and the early version of this work can
be seen in. Credit for this work goes to the team: [~vishwajeet.dusane],
[~snayak], [~srevanka], [~kiranch], [~chakrab], [~omkarksa], [~snvijaya],
[~ansaiprasanna] [~jsangwan]
h2. Test
Besides Contract tests, we have used ADL as the additional file system in the
current public preview release. Various different customer and test workloads
have been run against clusters with such configurations for quite some time.
The current version reflects to the version of the code tested and used in our
production environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
