[jira] [Commented] (HADOOP-18080) ABFS: Skip testEtagConsistencyAcrossRename for Non-HNS accounts
[ https://issues.apache.org/jira/browse/HADOOP-18080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477148#comment-17477148 ] Sumangala Patki commented on HADOOP-18080: -- Thanks for pointing to the fix! Yes etags are present, there was a mismatch causing the failure. I believe a copy of the file is created during rename, resulting in a new etag. > ABFS: Skip testEtagConsistencyAcrossRename for Non-HNS accounts > --- > > Key: HADOOP-18080 > URL: https://issues.apache.org/jira/browse/HADOOP-18080 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.2 >Reporter: Sumangala Patki >Priority: Major > > The rename operation is not supported for non-HNS accounts. Hence, tests > verifying matching etag for file before and after rename should not be run > against non-HNS accounts. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18080) ABFS: Skip testEtagConsistencyAcrossRename for Non-HNS accounts
Sumangala Patki created HADOOP-18080: Summary: ABFS: Skip testEtagConsistencyAcrossRename for Non-HNS accounts Key: HADOOP-18080 URL: https://issues.apache.org/jira/browse/HADOOP-18080 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.2 Reporter: Sumangala Patki The rename operation is not supported for non-HNS accounts. Hence, tests verifying matching etag for file before and after rename should not be run against non-HNS accounts. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18075) ABFS: Fix failure caused by listFiles() in ITestAbfsRestOperationException
Sumangala Patki created HADOOP-18075: Summary: ABFS: Fix failure caused by listFiles() in ITestAbfsRestOperationException Key: HADOOP-18075 URL: https://issues.apache.org/jira/browse/HADOOP-18075 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.2 Reporter: Sumangala Patki Assignee: Sumangala Patki testAbfsRestOperationExceptionFormat in ITestAbfsRestOperationException fails due to the wrong exception format of the FileNotFound exception. The test invokes the Filesystem method listFiles(), and the exception thrown is found to be of the GetPathStatus format instead of ListStatus (difference in number of error fields in response). The Filesystem implementation of listFiles() calls listLocatedStatus(), which then makes a listStatus call. A recent check-in that added implementation for listLocatedStatus() in ABFS driver included a GetFileStatus request before ListStatus api are invoked, leading to the aberrant FNF exception format. The fix eliminates the GetPathStatus request before ListStatus is called. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18075) ABFS: Fix failure caused by listFiles() in ITestAbfsRestOperationException
[ https://issues.apache.org/jira/browse/HADOOP-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-18075: - Description: testAbfsRestOperationExceptionFormat in ITestAbfsRestOperationException fails due to the wrong exception format of the FileNotFound exception. The test invokes the Filesystem method listFiles(), and the exception thrown is found to be of the GetPathStatus format instead of ListStatus (difference in number of error fields in response). The Filesystem implementation of listFiles() calls listLocatedStatus(), which then makes a listStatus call. A recent check-in that added implementation for listLocatedStatus() in ABFS driver led to a GetFileStatus request before ListStatus api are invoked, leading to the aberrant FNF exception format. The fix eliminates the GetPathStatus request before ListStatus is called. was: testAbfsRestOperationExceptionFormat in ITestAbfsRestOperationException fails due to the wrong exception format of the FileNotFound exception. The test invokes the Filesystem method listFiles(), and the exception thrown is found to be of the GetPathStatus format instead of ListStatus (difference in number of error fields in response). The Filesystem implementation of listFiles() calls listLocatedStatus(), which then makes a listStatus call. A recent check-in that added implementation for listLocatedStatus() in ABFS driver included a GetFileStatus request before ListStatus api are invoked, leading to the aberrant FNF exception format. The fix eliminates the GetPathStatus request before ListStatus is called. > ABFS: Fix failure caused by listFiles() in ITestAbfsRestOperationException > -- > > Key: HADOOP-18075 > URL: https://issues.apache.org/jira/browse/HADOOP-18075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.2 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > > testAbfsRestOperationExceptionFormat in ITestAbfsRestOperationException fails > due to the wrong exception format of the FileNotFound exception. The test > invokes the Filesystem method listFiles(), and the exception thrown is found > to be of the GetPathStatus format instead of ListStatus (difference in number > of error fields in response). > The Filesystem implementation of listFiles() calls listLocatedStatus(), which > then makes a listStatus call. A recent check-in that added implementation for > listLocatedStatus() in ABFS driver led to a GetFileStatus request before > ListStatus api are invoked, leading to the aberrant FNF exception format. The > fix eliminates the GetPathStatus request before ListStatus is called. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18071) ABFS: Set driver global timeout for ITestAzureBlobFileSystemBasics
Sumangala Patki created HADOOP-18071: Summary: ABFS: Set driver global timeout for ITestAzureBlobFileSystemBasics Key: HADOOP-18071 URL: https://issues.apache.org/jira/browse/HADOOP-18071 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.1 Reporter: Sumangala Patki Assignee: Sumangala Patki Unlike all other ABFS driver tests that have a timeout of 15min, ITestAzureBlobFileSystemBasics times out after 30s due to the global timeout inherited from FileSystemContractBaseTest. Setting a 15min timeout for the test will ensure sufficient time to allow retries and avoid transient failures. Example failure: testListOnFolderWithNoChildren(org.apache.hadoop.fs.azurebfs.contract.ITestAzureBlobFileSystemBasics) Time elapsed: 34.655 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 3 milliseconds -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17912) ABFS: Support for Encryption Context
Sumangala Patki created HADOOP-17912: Summary: ABFS: Support for Encryption Context Key: HADOOP-17912 URL: https://issues.apache.org/jira/browse/HADOOP-17912 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.1 Reporter: Sumangala Patki Assignee: Sumangala Patki Support for customer-provided encryption keys at the file level, superceding the global (account-level) key use in HADOOP-17536. ABFS driver will support an "EncryptionContext" plugin for retrieving encryption information, the implementation for which should be provided by the client. The keys/context retrieved will be sent via request headers to the server, which will store the encryption context. Subsequent REST calls to server that access data/user metadata of the file will require fetching the encryption context through a GetFileProperties call and retrieving the key from the custom provider, before sending the request. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17896) ABFS: Stabilize openFile withStatus
Sumangala Patki created HADOOP-17896: Summary: ABFS: Stabilize openFile withStatus Key: HADOOP-17896 URL: https://issues.apache.org/jira/browse/HADOOP-17896 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.1 Reporter: Sumangala Patki Add support for more FileStatus types at OpenFile, and address minor concerns with HADOOP-17682 PR. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17873) ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException
Sumangala Patki created HADOOP-17873: Summary: ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException Key: HADOOP-17873 URL: https://issues.apache.org/jira/browse/HADOOP-17873 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.1 Reporter: Sumangala Patki Assignee: Sumangala Patki To address transient failures in the following test classes: * ITestAbfsStreamStatistics: Uses a filesystem level instance to record read/write statistics, which also tracks these operations in other tests. running parallelly. To be marked for sequential run only to avoid transient failure * ITestAbfsRestOperationException: The use of a static member to track retry count causes transient failures when two tests of this class happen to run together. Switch to non-static variable for assertions on retry count -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17867) WASB: Fix Compiler Warnings
Sumangala Patki created HADOOP-17867: Summary: WASB: Fix Compiler Warnings Key: HADOOP-17867 URL: https://issues.apache.org/jira/browse/HADOOP-17867 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.1 Reporter: Sumangala Patki To track a list of WASB compiler warnings, which are also intermittently affecting yetus runs on ABFS driver PRs. # hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobOutputStream.java:210:29:[deprecation] HFLUSH in StreamCapabilities has been deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17863) ABFS: Fix compiler deprecation warning in TextFileBasedIdentityHandler
Sumangala Patki created HADOOP-17863: Summary: ABFS: Fix compiler deprecation warning in TextFileBasedIdentityHandler Key: HADOOP-17863 URL: https://issues.apache.org/jira/browse/HADOOP-17863 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.1 Reporter: Sumangala Patki Assignee: Sumangala Patki TextFileBasedIdentityHandler uses an instance of LineIterator whose closeQuietly method has been deprecated, resulting in compiler warnings during yetus runs. Fix by leveraging a try-with-resources block to avoid the explicit call to closeQuietly by the LineIterator instance. hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TextFileBasedIdentityHandler.java:192:18:[deprecation] closeQuietly(LineIterator) in LineIterator has been deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17862) ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator
Sumangala Patki created HADOOP-17862: Summary: ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator Key: HADOOP-17862 URL: https://issues.apache.org/jira/browse/HADOOP-17862 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.1 Reporter: Sumangala Patki Assignee: Sumangala Patki Hadoop yetus run shows a java compiler warning for unchecked casting of Object to Iterator in a method of AbfsListStatusRemoteIterator class. This can be resolved by introducing a new class to hold the iterator and exception thrown when applicable. hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsListStatusRemoteIterator.java:97:38:[unchecked] unchecked cast This task will also add logging to facilitate better investigation of the transient failure tracked in HADOOP-17797. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17826) ABFS: Transient failure of TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting
[ https://issues.apache.org/jira/browse/HADOOP-17826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-17826: - Description: Transient failure of the below test observed for HNS OAuth, AppendBlob HNS OAuth and Non-HNS SharedKey combinations. The value denoted by "actual value" below varies across failures, and exceeds the upper limit of the expected range. _TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 10 is not within the expected range: [5.60, 8.40]._ Verified failure with client and server in the same region to rule out network issues. was: Transient failure of the below test observed for HNS OAuth, AppendBlob HNS OAuth and Non-HNS SharedKey combinations. The value denoted by "actual value" below varies across failures, and exceeds the upper limit of the expected range. _TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 10 is not within the expected range: [5.60, 8.40]._ > ABFS: Transient failure of > TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting > -- > > Key: HADOOP-17826 > URL: https://issues.apache.org/jira/browse/HADOOP-17826 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0 >Reporter: Sumangala Patki >Priority: Major > > Transient failure of the below test observed for HNS OAuth, AppendBlob HNS > OAuth and Non-HNS SharedKey combinations. The value denoted by "actual value" > below varies across failures, and exceeds the upper limit of the expected > range. > _TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 > The actual value 10 is not within the expected range: [5.60, 8.40]._ > Verified failure with client and server in the same region to rule out > network issues. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17826) ABFS: Transient failure of TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting
Sumangala Patki created HADOOP-17826: Summary: ABFS: Transient failure of TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting Key: HADOOP-17826 URL: https://issues.apache.org/jira/browse/HADOOP-17826 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.4.0 Reporter: Sumangala Patki Transient failure of the below test observed for HNS OAuth, AppendBlob HNS OAuth and Non-HNS SharedKey combinations. The value denoted by "actual value" below varies across failures, and exceeds the upper limit of the expected range. _TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 10 is not within the expected range: [5.60, 8.40]._ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17790) ABFS: Modifications to TracingContext
Sumangala Patki created HADOOP-17790: Summary: ABFS: Modifications to TracingContext Key: HADOOP-17790 URL: https://issues.apache.org/jira/browse/HADOOP-17790 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.1 Reporter: Sumangala Patki TracingContext introduced in the PR [HADOOP-17290|https://github.com/apache/hadoop/pull/2520] tracks a request with identifiers as it passes through ABFS layers. Creating this Jira to track suggested modifications and improvements to the tracing structure. * rename Listener; remove it from TracingContext constructor, confine to setter * make fields (identifiers) immutable; eliminate retry count field => reduce instance cloning * introduce child classes of TracingContext to handle stream/continuation ops -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17781) ABFS: ITestAzureBlobFileSystemLease test failure for Appendblob HNS OAuth
Sumangala Patki created HADOOP-17781: Summary: ABFS: ITestAzureBlobFileSystemLease test failure for Appendblob HNS OAuth Key: HADOOP-17781 URL: https://issues.apache.org/jira/browse/HADOOP-17781 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.1 Reporter: Sumangala Patki Test testTwoWritersCreateAppendNoInfiniteLease fails with error: java.io.IOException: Operation failed: "Value for one of the query parameters specified in the request URI is invalid." -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17765) ABFS: Use Unique File Paths in Tests
Sumangala Patki created HADOOP-17765: Summary: ABFS: Use Unique File Paths in Tests Key: HADOOP-17765 URL: https://issues.apache.org/jira/browse/HADOOP-17765 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.1 Reporter: Sumangala Patki Assignee: Sumangala Patki Many of ABFS driver tests use common names for file paths (e.g., "/testfile"). This poses a risk of errors during parallel test runs when static variables (such as those for monitoring stats) affected by file paths are introduced. Using unique test file names will avoid possible errors arising from shared resources during parallel runs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17682) ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
[ https://issues.apache.org/jira/browse/HADOOP-17682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-17682: - Description: ABFS open methods require certain information (contentLength, eTag, etc) to to create an InputStream for the file at the given path. This information is retrieved via a GetFileStatus request to backend. However, client applications may often have access to the FileStatus prior to invoking the open API. Providing this FileStatus to the driver through the OpenFileParameters argument of openFileWithOptions() can help avoid the call to Store for FileStatus. This PR adds handling for the FileStatus instance (if any) provided via the OpenFileParameters argument. was: ABFS open methods require certain information (contentLength, eTag, etc) to be sent to Store to open the file at the given path. This information is retrieved via a GetFileStatus request to backend, resulting in a total of two Store calls to carry out the open function. However, client applications may often have access to the FileStatus prior to invoking the open API. Providing this FileStatus to the driver through the OpenFileParameters argument of openFileWithOptions() can avoid the additional call to Store for FileStatus. This PR adds handling for the FileStatus instance (if any) provided via the OpenFileParameters argument. > ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters > -- > > Key: HADOOP-17682 > URL: https://issues.apache.org/jira/browse/HADOOP-17682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > ABFS open methods require certain information (contentLength, eTag, etc) to > to create an InputStream for the file at the given path. This information is > retrieved via a GetFileStatus request to backend. > However, client applications may often have access to the FileStatus prior to > invoking the open API. Providing this FileStatus to the driver through the > OpenFileParameters argument of openFileWithOptions() can help avoid the call > to Store for FileStatus. > This PR adds handling for the FileStatus instance (if any) provided via the > OpenFileParameters argument. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17682) ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
Sumangala Patki created HADOOP-17682: Summary: ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters Key: HADOOP-17682 URL: https://issues.apache.org/jira/browse/HADOOP-17682 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Reporter: Sumangala Patki Assignee: Sumangala Patki ABFS open methods require certain information (contentLength, eTag, etc) to be sent to Store to open the file at the given path. This information is retrieved via a GetFileStatus request to backend, resulting in a total of two Store calls to carry out the open function. However, client applications may often have access to the FileStatus prior to invoking the open API. Providing this FileStatus to the driver through the OpenFileParameters argument of openFileWithOptions() can avoid the additional call to Store for FileStatus. This PR adds handling for the FileStatus instance (if any) provided via the OpenFileParameters argument. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17618) ABFS: Partially obfuscate SAS object IDs in Logs
[ https://issues.apache.org/jira/browse/HADOOP-17618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-17618: - Description: Delegation SAS tokens are created using various parameters for specifying details such as permissions and validity. The requests are logged, along with values of all the query parameters. This change will partially mask values logged for the following object IDs representing the security principal: skoid, saoid, suoid (was: Delegation SAS tokens are created using various parameters for specifying details such as permissions and validity. The requests are logged, along with values of all the query parameters. This change will partially mask the values of the following object IDs representing the security principal: skoid, saoid, suoid) > ABFS: Partially obfuscate SAS object IDs in Logs > > > Key: HADOOP-17618 > URL: https://issues.apache.org/jira/browse/HADOOP-17618 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Delegation SAS tokens are created using various parameters for specifying > details such as permissions and validity. The requests are logged, along with > values of all the query parameters. This change will partially mask values > logged for the following object IDs representing the security principal: > skoid, saoid, suoid -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17618) ABFS: Partially obfuscate SAS object IDs in Logs
[ https://issues.apache.org/jira/browse/HADOOP-17618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki reassigned HADOOP-17618: Assignee: Sumangala Patki > ABFS: Partially obfuscate SAS object IDs in Logs > > > Key: HADOOP-17618 > URL: https://issues.apache.org/jira/browse/HADOOP-17618 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > > Delegation SAS tokens are created using various parameters for specifying > details such as permissions and validity. The requests are logged, along with > values of all the query parameters. This change will partially mask the > values of the following object IDs representing the security principal: > skoid, saoid, suoid -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17618) ABFS: Partially obfuscate SAS object IDs in Logs
Sumangala Patki created HADOOP-17618: Summary: ABFS: Partially obfuscate SAS object IDs in Logs Key: HADOOP-17618 URL: https://issues.apache.org/jira/browse/HADOOP-17618 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.1 Reporter: Sumangala Patki Delegation SAS tokens are created using various parameters for specifying details such as permissions and validity. The requests are logged, along with values of all the query parameters. This change will partially mask the values of the following object IDs representing the security principal: skoid, saoid, suoid -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
[ https://issues.apache.org/jira/browse/HADOOP-17596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki reassigned HADOOP-17596: Assignee: Sumangala Patki > ABFS: Change default Readahead Queue Depth from num(processors) to const > > > Key: HADOOP-17596 > URL: https://issues.apache.org/jira/browse/HADOOP-17596 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The default value of readahead queue depth is currently set to the number of > available processors. However, this can result in one inputstream instance > consuming more processor time. To ensure equal thread allocation during read > for all inputstreams created in a session, we change the default readahead > queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17596) ABFS: Change default Readahead Queue Depth from num(processors) to const
Sumangala Patki created HADOOP-17596: Summary: ABFS: Change default Readahead Queue Depth from num(processors) to const Key: HADOOP-17596 URL: https://issues.apache.org/jira/browse/HADOOP-17596 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.1 Reporter: Sumangala Patki The default value of readahead queue depth is currently set to the number of available processors. However, this can result in one inputstream instance consuming more processor time. To ensure equal thread allocation during read for all inputstreams created in a session, we change the default readahead queue depth to a constant (2). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17428) ABFS: Implementation for getContentSummary
[ https://issues.apache.org/jira/browse/HADOOP-17428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304706#comment-17304706 ] Sumangala Patki commented on HADOOP-17428: -- ABFS is currently doing a breadth-first search in parallel. This implementation is along the lines of the existing ADLS Gen1 implementation for the method. We believe this parallelized bfs approach will suffice for the present usage > ABFS: Implementation for getContentSummary > -- > > Key: HADOOP-17428 > URL: https://issues.apache.org/jira/browse/HADOOP-17428 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > > Adds implementation for HDFS method getContentSummary, which takes in a Path > argument and returns details such as file/directory count and space utilized > under that path. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17503) ABFS: Fix auth failures getting counted as throttling errors in ABFS throttling interceptor
[ https://issues.apache.org/jira/browse/HADOOP-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki resolved HADOOP-17503. -- Resolution: Duplicate Implemented in HADOOP-17576 > ABFS: Fix auth failures getting counted as throttling errors in ABFS > throttling interceptor > --- > > Key: HADOOP-17503 > URL: https://issues.apache.org/jira/browse/HADOOP-17503 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0 >Reporter: Bilahari T H >Priority: Minor > > Currently the throttling intercepter counts the auth failures against > throttling errors. This needs to be fixed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17265) ABFS: Support for Client Correlation ID
[ https://issues.apache.org/jira/browse/HADOOP-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki resolved HADOOP-17265. -- Resolution: Abandoned > ABFS: Support for Client Correlation ID > --- > > Key: HADOOP-17265 > URL: https://issues.apache.org/jira/browse/HADOOP-17265 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sumangala Patki >Priority: Major > Labels: abfsactive, pull-request-available > Time Spent: 3h 40m > Remaining Estimate: 0h > > Introducing a client correlation ID that appears in the Azure Storage > diagnostic logs. This will modify the client request header to include the > client-provided identifier for correlating requests. A valid correlation ID > comprises alphanumeric characters and/or hyphens, and can be up to 72 > characters in length. Invalid or missing entries for this configuration are > substituted with the empty string default. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17290) ABFS: Add Identifiers to Client Request Header
[ https://issues.apache.org/jira/browse/HADOOP-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki reassigned HADOOP-17290: Assignee: Sumangala Patki > ABFS: Add Identifiers to Client Request Header > -- > > Key: HADOOP-17290 > URL: https://issues.apache.org/jira/browse/HADOOP-17290 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: abfsactive, pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > Adding unique values to the client request header to assist in correlating > requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17428) ABFS: Implementation for getContentSummary
[ https://issues.apache.org/jira/browse/HADOOP-17428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki reassigned HADOOP-17428: Assignee: Sumangala Patki > ABFS: Implementation for getContentSummary > -- > > Key: HADOOP-17428 > URL: https://issues.apache.org/jira/browse/HADOOP-17428 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > > Adds implementation for HDFS method getContentSummary, which takes in a Path > argument and returns details such as file/directory count and space utilized > under that path. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17537) Correct abfs test assertion reversed in HADOOP-13327
[ https://issues.apache.org/jira/browse/HADOOP-17537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki resolved HADOOP-17537. -- Resolution: Fixed > Correct abfs test assertion reversed in HADOOP-13327 > > > Key: HADOOP-17537 > URL: https://issues.apache.org/jira/browse/HADOOP-17537 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, test >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 40m > Remaining Estimate: 0h > > HADOOP-13327 introduces, among other changes, functions to simplify assert > for checking stream capabilities. This PR fixes (originally) assertFalse > statements whose logic has been reversed to assertTrue when the above change > was checked-in. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17527) ABFS: Fix boundary conditions in InputStream seek and skip
[ https://issues.apache.org/jira/browse/HADOOP-17527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki resolved HADOOP-17527. -- Resolution: Won't Fix > ABFS: Fix boundary conditions in InputStream seek and skip > -- > > Key: HADOOP-17527 > URL: https://issues.apache.org/jira/browse/HADOOP-17527 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 4h 50m > Remaining Estimate: 0h > > Modify AbfsInputStream seek method to throw EOF exception on seek to > contentLength for a non-empty file. With this change, it will no longer be > possible for the inputstream position (as obtained by getPos() API) to be > moved to contentlength manually, except post reading the last byte. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17576) ABFS: Disable throttling update for auth failures
[ https://issues.apache.org/jira/browse/HADOOP-17576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki reassigned HADOOP-17576: Assignee: Sumangala Patki > ABFS: Disable throttling update for auth failures > - > > Key: HADOOP-17576 > URL: https://issues.apache.org/jira/browse/HADOOP-17576 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Throttling metrics are updated post the execution of each request. Failures > related to fetching access tokens and signing requests do not occur at the > Store. Hence, such operations should not contribute to the measured Store > failures, and are therefore excluded from the metric update for throttling. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17576) ABFS: Disable throttling update for auth failures
[ https://issues.apache.org/jira/browse/HADOOP-17576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki reassigned HADOOP-17576: Assignee: (was: Sumangala Patki) > ABFS: Disable throttling update for auth failures > - > > Key: HADOOP-17576 > URL: https://issues.apache.org/jira/browse/HADOOP-17576 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Throttling metrics are updated post the execution of each request. Failures > related to fetching access tokens and signing requests do not occur at the > Store. Hence, such operations should not contribute to the measured Store > failures, and are therefore excluded from the metric update for throttling. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17576) ABFS: Disable throttling update for auth failures
[ https://issues.apache.org/jira/browse/HADOOP-17576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki reassigned HADOOP-17576: Assignee: Sumangala Patki > ABFS: Disable throttling update for auth failures > - > > Key: HADOOP-17576 > URL: https://issues.apache.org/jira/browse/HADOOP-17576 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Throttling metrics are updated post the execution of each request. Failures > related to fetching access tokens and signing requests do not occur at the > Store. Hence, such operations should not contribute to the measured Store > failures, and are therefore excluded from the metric update for throttling. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17576) ABFS: Disable throttling update for auth failures
Sumangala Patki created HADOOP-17576: Summary: ABFS: Disable throttling update for auth failures Key: HADOOP-17576 URL: https://issues.apache.org/jira/browse/HADOOP-17576 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.1 Reporter: Sumangala Patki Throttling metrics are updated post the execution of each request. Failures related to fetching access tokens and signing requests do not occur at the Store. Hence, such operations should not contribute to the measured Store failures, and are therefore excluded from the metric update for throttling. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17548) ABFS: Config for Mkdir overwrite
[ https://issues.apache.org/jira/browse/HADOOP-17548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-17548: - Description: The call to mkdirs with overwrite set to true results in an additional call to set properties (LMT update, etc) at the backend, which is not required for the HDFS scenario. Moreover, mkdirs on an existing file path returns success. This PR provides an option to set the overwrite parameter to false, and ensures that mkdirs on a file throws an exception. (was: The call to mkdirs with overwrite set to true results in an overhead set property call (LMT update, etc), which is unnecessary. Moreover, mkdirs on an existing file path returns success. This PR provides an option to set the overwrite parameter to false, and ensures that mkdirs on a file throws an exception.) > ABFS: Config for Mkdir overwrite > > > Key: HADOOP-17548 > URL: https://issues.apache.org/jira/browse/HADOOP-17548 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Priority: Major > > The call to mkdirs with overwrite set to true results in an additional call > to set properties (LMT update, etc) at the backend, which is not required for > the HDFS scenario. Moreover, mkdirs on an existing file path returns success. > This PR provides an option to set the overwrite parameter to false, and > ensures that mkdirs on a file throws an exception. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17548) ABFS: Config for Mkdir overwrite
[ https://issues.apache.org/jira/browse/HADOOP-17548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-17548: - Summary: ABFS: Config for Mkdir overwrite (was: ABFS: Change mkdirs default overwrite to false) > ABFS: Config for Mkdir overwrite > > > Key: HADOOP-17548 > URL: https://issues.apache.org/jira/browse/HADOOP-17548 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Priority: Major > > The call to mkdirs with overwrite set to true results in an overhead set > property call (LMT update, etc), which is unnecessary. Moreover, mkdirs on an > existing file path returns success. This PR provides an option to set the > overwrite parameter to false, and ensures that mkdirs on a file throws an > exception. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17548) ABFS: Change mkdirs default overwrite to false
[ https://issues.apache.org/jira/browse/HADOOP-17548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-17548: - Description: The call to mkdirs with overwrite set to true results in an overhead set property call (LMT update, etc), which is unnecessary. Moreover, mkdirs on an existing file path returns success. This PR provides an option to set the overwrite parameter to false, and ensures that mkdirs on a file throws an exception. (was: The call to mkdirs with overwrite set to true results in an overhead set property call (LMT update, etc), which is unnecessary. This change sets the overwrite parameter to false and adds the required checks to return success as for the overwrite=true case.) > ABFS: Change mkdirs default overwrite to false > -- > > Key: HADOOP-17548 > URL: https://issues.apache.org/jira/browse/HADOOP-17548 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Priority: Major > > The call to mkdirs with overwrite set to true results in an overhead set > property call (LMT update, etc), which is unnecessary. Moreover, mkdirs on an > existing file path returns success. This PR provides an option to set the > overwrite parameter to false, and ensures that mkdirs on a file throws an > exception. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17548) ABFS: Change mkdirs default overwrite to false
Sumangala Patki created HADOOP-17548: Summary: ABFS: Change mkdirs default overwrite to false Key: HADOOP-17548 URL: https://issues.apache.org/jira/browse/HADOOP-17548 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.3.1 Reporter: Sumangala Patki The call to mkdirs with overwrite set to true results in an overhead set property call (LMT update, etc), which is unnecessary. This change sets the overwrite parameter to false and adds the required checks to return success as for the overwrite=true case. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17537) Correct assertion reversed in HADOOP-13327
[ https://issues.apache.org/jira/browse/HADOOP-17537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-17537: - Description: HADOOP-13327 introduces, among other changes, functions to simplify assert for checking stream capabilities. This PR fixes (originally) assertFalse statements whose logic has been reversed to assertTrue when the above change was checked-in. (was: HADOOP-13327 introduces, among other changes, functions to simplify assert for checking stream capabilities. This PR fixes (originally) assertFalse statements whose logic has been reversed to assertTrue when the above change was checked-in.) > Correct assertion reversed in HADOOP-13327 > -- > > Key: HADOOP-17537 > URL: https://issues.apache.org/jira/browse/HADOOP-17537 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HADOOP-13327 introduces, among other changes, functions to simplify assert > for checking stream capabilities. This PR fixes (originally) assertFalse > statements whose logic has been reversed to assertTrue when the above change > was checked-in. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17537) Correct assertion reversed in HADOOP-13327
Sumangala Patki created HADOOP-17537: Summary: Correct assertion reversed in HADOOP-13327 Key: HADOOP-17537 URL: https://issues.apache.org/jira/browse/HADOOP-17537 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.0 Reporter: Sumangala Patki HADOOP-13327 introduces, among other changes, functions to simplify assert for checking stream capabilities. This PR fixes (originally) assertFalse statements whose logic has been reversed to assertTrue when the above change was checked-in. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17527) ABFS: Fix boundary conditions in InputStream seek and skip
[ https://issues.apache.org/jira/browse/HADOOP-17527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-17527: - Description: Modify AbfsInputStream seek method to throw EOF exception on seek to contentLength for a non-empty file. With this change, it will no longer be possible for the inputstream position (as obtained by getPos() API) to be moved to contentlength manually, except post reading the last byte. (was: Modify AbfsInputStream seek method to throw EOF exception on seek to contentLength for a non-empty file) > ABFS: Fix boundary conditions in InputStream seek and skip > -- > > Key: HADOOP-17527 > URL: https://issues.apache.org/jira/browse/HADOOP-17527 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > Modify AbfsInputStream seek method to throw EOF exception on seek to > contentLength for a non-empty file. With this change, it will no longer be > possible for the inputstream position (as obtained by getPos() API) to be > moved to contentlength manually, except post reading the last byte. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17527) ABFS: Fix boundary conditions in InputStream seek and skip
[ https://issues.apache.org/jira/browse/HADOOP-17527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-17527: - Description: Modify AbfsInputStream seek method to throw EOF exception on seek to contentLength for a non-empty file (was: Fix bug in condition for validating position in AbfsInputStream seek method) > ABFS: Fix boundary conditions in InputStream seek and skip > -- > > Key: HADOOP-17527 > URL: https://issues.apache.org/jira/browse/HADOOP-17527 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > Modify AbfsInputStream seek method to throw EOF exception on seek to > contentLength for a non-empty file -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17527) ABFS: Fix boundary conditions in InputStream seek and skip
[ https://issues.apache.org/jira/browse/HADOOP-17527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-17527: - Summary: ABFS: Fix boundary conditions in InputStream seek and skip (was: ABFS: Fix condition in InputStream seek) > ABFS: Fix boundary conditions in InputStream seek and skip > -- > > Key: HADOOP-17527 > URL: https://issues.apache.org/jira/browse/HADOOP-17527 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Fix bug in condition for validating position in AbfsInputStream seek method -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17527) ABFS: Fix condition in InputStream seek
Sumangala Patki created HADOOP-17527: Summary: ABFS: Fix condition in InputStream seek Key: HADOOP-17527 URL: https://issues.apache.org/jira/browse/HADOOP-17527 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.0 Reporter: Sumangala Patki Fix bug in condition for validating position in AbfsInputStream seek method -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17428) ABFS: Implementation for getContentSummary
Sumangala Patki created HADOOP-17428: Summary: ABFS: Implementation for getContentSummary Key: HADOOP-17428 URL: https://issues.apache.org/jira/browse/HADOOP-17428 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.0 Reporter: Sumangala Patki Adds implementation for HDFS method getContentSummary, which takes in a Path argument and returns details such as file/directory count and space utilized under that path. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17422) ABFS: Set default ListMaxResults to max server limit
[
https://issues.apache.org/jira/browse/HADOOP-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumangala Patki updated HADOOP-17422:
-
Description: {{Changing the default value of maximum size of }}{{results to
be}}{{ returned by ListStatus from 500 to 5000, since the maximum number of
items supported by a listStatus server call is 5000.}} (was: {{Changing value
of DEFAULT_AZURE_LIST_MAX_RESULTS config (number of items listed) from 500 to
5000, since the maximum number of items supported by a listPath server call is
5000.}})
> ABFS: Set default ListMaxResults to max server limit
>
>
> Key: HADOOP-17422
> URL: https://issues.apache.org/jira/browse/HADOOP-17422
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Sumangala Patki
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{Changing the default value of maximum size of }}{{results to be}}{{
> returned by ListStatus from 500 to 5000, since the maximum number of items
> supported by a listStatus server call is 5000.}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17422) ABFS: Set default ListMaxResults to max server limit
[
https://issues.apache.org/jira/browse/HADOOP-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumangala Patki updated HADOOP-17422:
-
Description: {{Changing value of DEFAULT_AZURE_LIST_MAX_RESULTS config
(number of items listed) from 500 to 5000, since the maximum number of items
supported by a listPath server call is 5000.}} (was: {{Changing value of
DEFAULT_AZURE_LIST_MAX_RESULTS config (number of items listed) from 500 to
5000, which is the same as the maximum number of items supported by a listPath
server call.}})
> ABFS: Set default ListMaxResults to max server limit
>
>
> Key: HADOOP-17422
> URL: https://issues.apache.org/jira/browse/HADOOP-17422
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Sumangala Patki
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{Changing value of DEFAULT_AZURE_LIST_MAX_RESULTS config (number of items
> listed) from 500 to 5000, since the maximum number of items supported by a
> listPath server call is 5000.}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17422) ABFS: Set default ListMaxResults to max server limit
[
https://issues.apache.org/jira/browse/HADOOP-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumangala Patki updated HADOOP-17422:
-
Description: {{Changing value of DEFAULT_AZURE_LIST_MAX_RESULTS config
(number of items listed) from 500 to 5000, which is the same as the maximum
number of items supported by a listPath server call.}} (was: {{Changing value
of DEFAULT_AZURE_LIST_MAX_RESULTS config to 5000, which is the same as the
maximum number of items supported by a listPath server call.}})
> ABFS: Set default ListMaxResults to max server limit
>
>
> Key: HADOOP-17422
> URL: https://issues.apache.org/jira/browse/HADOOP-17422
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Sumangala Patki
>Priority: Major
>
> {{Changing value of DEFAULT_AZURE_LIST_MAX_RESULTS config (number of items
> listed) from 500 to 5000, which is the same as the maximum number of items
> supported by a listPath server call.}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17422) ABFS: Set default ListMaxResults to max server limit
Sumangala Patki created HADOOP-17422:
Summary: ABFS: Set default ListMaxResults to max server limit
Key: HADOOP-17422
URL: https://issues.apache.org/jira/browse/HADOOP-17422
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/azure
Affects Versions: 3.3.0
Reporter: Sumangala Patki
{{Changing value of DEFAULT_AZURE_LIST_MAX_RESULTS config to 5000, which is the
same as the maximum number of items supported by a listPath server call.}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17265) ABFS: Support for Client Correlation ID
[ https://issues.apache.org/jira/browse/HADOOP-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-17265: - Description: Introducing a client correlation ID that appears in the Azure Storage diagnostic logs. This will modify the client request header to include the client-provided identifier for correlating requests. A valid correlation ID comprises alphanumeric characters and/or hyphens, and can be up to 72 characters in length. Invalid or missing entries for this configuration are substituted with the empty string default. (was: Introducing a client correlation ID that appears in the Azure diagnostic logs) > ABFS: Support for Client Correlation ID > --- > > Key: HADOOP-17265 > URL: https://issues.apache.org/jira/browse/HADOOP-17265 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sumangala Patki >Priority: Major > Labels: abfsactive, pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Introducing a client correlation ID that appears in the Azure Storage > diagnostic logs. This will modify the client request header to include the > client-provided identifier for correlating requests. A valid correlation ID > comprises alphanumeric characters and/or hyphens, and can be up to 72 > characters in length. Invalid or missing entries for this configuration are > substituted with the empty string default. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17290) ABFS: Add Identifiers to Client Request Header
Sumangala Patki created HADOOP-17290: Summary: ABFS: Add Identifiers to Client Request Header Key: HADOOP-17290 URL: https://issues.apache.org/jira/browse/HADOOP-17290 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.0 Reporter: Sumangala Patki Adding unique values to the client request header to assist in correlating requests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17265) ABFS: Support for Client Correlation ID
[ https://issues.apache.org/jira/browse/HADOOP-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198424#comment-17198424 ] Sumangala Patki commented on HADOOP-17265: -- Here's a link to the docs on logging- [https://docs.microsoft.com/en-us/azure/storage/common/storage-analytics-logging?tabs=dotnet]. The valid ID structure will be defined by the PR code and the ID will be available in the azure storage diagnostic logs > ABFS: Support for Client Correlation ID > --- > > Key: HADOOP-17265 > URL: https://issues.apache.org/jira/browse/HADOOP-17265 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sumangala Patki >Priority: Major > Labels: abfsactive > > Introducing a client correlation ID that appears in the Azure diagnostic logs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17265) ABFS: Support for Client Correlation ID
[ https://issues.apache.org/jira/browse/HADOOP-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumangala Patki updated HADOOP-17265: - Labels: abfsactive (was: ) > ABFS: Support for Client Correlation ID > --- > > Key: HADOOP-17265 > URL: https://issues.apache.org/jira/browse/HADOOP-17265 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sumangala Patki >Priority: Major > Labels: abfsactive > > Introducing a client correlation ID that appears in the Azure diagnostic logs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17265) ABFS: Support for Client Correlation ID
Sumangala Patki created HADOOP-17265: Summary: ABFS: Support for Client Correlation ID Key: HADOOP-17265 URL: https://issues.apache.org/jira/browse/HADOOP-17265 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.0 Reporter: Sumangala Patki Introducing a client correlation ID that appears in the Azure diagnostic logs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
