[GitHub] [hadoop] steveloughran merged pull request #2264: HADOOP-17229. No updation of bytes received counter value after response failure occurs in ABFS
steveloughran merged pull request #2264: URL: https://github.com/apache/hadoop/pull/2264 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2264: HADOOP-17229. No updation of bytes received counter value after response failure occurs in ABFS
steveloughran commented on pull request #2264: URL: https://github.com/apache/hadoop/pull/2264#issuecomment-688736659 +1, merged to trunk, do a retest on branch-3.3 and I'll merge there too This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17229) Test failure as failed request body counted in byte received metric - ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
[
https://issues.apache.org/jira/browse/HADOOP-17229?focusedWorklogId=479960=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479960
]
ASF GitHub Bot logged work on HADOOP-17229:
---
Author: ASF GitHub Bot
Created on: 08/Sep/20 09:14
Start Date: 08/Sep/20 09:14
Worklog Time Spent: 10m
Work Description: steveloughran merged pull request #2264:
URL: https://github.com/apache/hadoop/pull/2264
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479960)
Time Spent: 1h 50m (was: 1h 40m)
> Test failure as failed request body counted in byte received metric -
> ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
> ---
>
> Key: HADOOP-17229
> URL: https://issues.apache.org/jira/browse/HADOOP-17229
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, test
>Affects Versions: 3.3.0
>Reporter: Sneha Vijayarajan
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: abfsactive, pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Bytes received counter increments for every request response received.
> [https://github.com/apache/hadoop/blob/d23cc9d85d887f01d72180bdf1af87dfdee15c5a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java#L251]
> This increments even for failed requests.
> Observed during testing done for HADOOP-17215. A request failed with 409
> Conflict contains response body as below:
> {"error":\{"code":"PathAlreadyExists","message":"The specified path already
> exists.\nRequestId:c3b2c55c-b01f-0061-7b31-7b6ee300\nTime:2020-08-25T22:44:07.2356054Z"}}
> The error body of 168 size is incremented in bytes_received counter.
> This also breaks the testcase testAbfsHttpResponseStatistics.
> {code:java}
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR] Tests run: 2,
> Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR]
> testAbfsHttpResponseStatistics(org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics)
> Time elapsed: 13.183 s <<< FAILURE!java.lang.AssertionError: Mismatch in
> bytes_received expected:<143> but was:<311> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.assertAbfsStatistics(AbstractAbfsIntegrationTest.java:445)
> at
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics.testAbfsHttpResponseStatistics(ITestAbfsNetworkStatistics.java:291)
> {code}
> [~mehakmeetSingh] - is the bytes_received counter increment for failed
> requests an expected behaviour ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] BilwaST commented on pull request #2239: YARN-10403. Add success/failure message to RMAdmin CLI queue refresh
BilwaST commented on pull request #2239: URL: https://github.com/apache/hadoop/pull/2239#issuecomment-688719152 Thanks @brucesw for updating. +1(Non binding) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] 1996fanrui commented on pull request #2241: HADOOP-17222. Create socket address combined with cache
1996fanrui commented on pull request #2241: URL: https://github.com/apache/hadoop/pull/2241#issuecomment-688674895 > I'm now on vacation. I will merge it this week. ok, thank you very much. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17222) Create socket address combined with cache to speed up hdfs client choose DataNode
[
https://issues.apache.org/jira/browse/HADOOP-17222?focusedWorklogId=479887=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479887
]
ASF GitHub Bot logged work on HADOOP-17222:
---
Author: ASF GitHub Bot
Created on: 08/Sep/20 07:25
Start Date: 08/Sep/20 07:25
Worklog Time Spent: 10m
Work Description: 1996fanrui commented on pull request #2241:
URL: https://github.com/apache/hadoop/pull/2241#issuecomment-688674895
> I'm now on vacation. I will merge it this week.
ok, thank you very much.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479887)
Time Spent: 40m (was: 0.5h)
> Create socket address combined with cache to speed up hdfs client choose
> DataNode
> -
>
> Key: HADOOP-17222
> URL: https://issues.apache.org/jira/browse/HADOOP-17222
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, hdfs-client
> Environment: HBase version: 2.1.0
> JVM: -Xmx2g -Xms2g
> hadoop hdfs version: 2.7.4
> disk:SSD
> OS:CentOS Linux release 7.4.1708 (Core)
> JMH Benchmark: @Fork(value = 1)
> @Warmup(iterations = 300)
> @Measurement(iterations = 300)
>Reporter: fanrui
>Assignee: fanrui
>Priority: Major
> Labels: pull-request-available
> Attachments: After Optimization remark.png, After optimization.svg,
> Before Optimization remark.png, Before optimization.svg
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Note：Not only the hdfs client can get the current benefit, all callers of
> NetUtils.createSocketAddr will get the benefit. Just use hdfs client as an
> example.
>
> Hdfs client selects best DN for hdfs Block. method call stack:
> DFSInputStream.chooseDataNode -> getBestNodeDNAddrPair ->
> NetUtils.createSocketAddr
> NetUtils.createSocketAddr creates the corresponding InetSocketAddress based
> on the host and port. There are some heavier operations in the
> NetUtils.createSocketAddr method, for example: URI.create(target), so
> NetUtils.createSocketAddr takes more time to execute.
> The following is my performance report. The report is based on HBase calling
> hdfs. HBase is a high-frequency access client for hdfs, because HBase read
> operations often access a small DataBlock (about 64k) instead of the entire
> HFile. In the case of high frequency access, the NetUtils.createSocketAddr
> method is time-consuming.
> h3. Test Environment：
>
> {code:java}
> HBase version: 2.1.0
> JVM: -Xmx2g -Xms2g
> hadoop hdfs version: 2.7.4
> disk:SSD
> OS:CentOS Linux release 7.4.1708 (Core)
> JMH Benchmark: @Fork(value = 1)
> @Warmup(iterations = 300)
> @Measurement(iterations = 300)
> {code}
> h4. Before Optimization FlameGraph:
> In the figure, we can see that DFSInputStream.getBestNodeDNAddrPair accounts
> for 4.86% of the entire CPU, and the creation of URIs accounts for a larger
> proportion.
> !Before Optimization remark.png!
> h3. Optimization ideas:
> NetUtils.createSocketAddr creates InetSocketAddress based on host and port.
> Here we can add Cache to InetSocketAddress. The key of Cache is host and
> port, and the value is InetSocketAddress.
> h4. After Optimization FlameGraph:
> In the figure, we can see that DFSInputStream.getBestNodeDNAddrPair accounts
> for 0.54% of the entire CPU. Here, ConcurrentHashMap is used as the Cache,
> and the ConcurrentHashMap.get() method gets data from the Cache. The CPU
> usage of DFSInputStream.getBestNodeDNAddrPair has been optimized from 4.86%
> to 0.54%.
> !After Optimization remark.png!
> h3. Original FlameGraph link:
> [Before
> Optimization|https://drive.google.com/file/d/133L5m75u2tu_KgKfGHZLEUzGR0XAfUl6/view?usp=sharing]
> [After Optimization
> FlameGraph|https://drive.google.com/file/d/133L5m75u2tu_KgKfGHZLEUzGR0XAfUl6/view?usp=sharing]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu opened a new pull request #2281: HDFS-15516.Add info for create flags in NameNode audit logs.
jianghuazhu opened a new pull request #2281: URL: https://github.com/apache/hadoop/pull/2281 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2272: HADOOP-17158. Test timeout for ITestAbfsInputStreamStatistics#testReadAheadCounters
steveloughran commented on pull request #2272: URL: https://github.com/apache/hadoop/pull/2272#issuecomment-688734784 merged to trunk; if you want it in branch-3.3 do a cherrypick, retest and let me know it worked This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17158) Test timeout for ITestAbfsInputStreamStatistics#testReadAheadCounters
[
https://issues.apache.org/jira/browse/HADOOP-17158?focusedWorklogId=479958=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479958
]
ASF GitHub Bot logged work on HADOOP-17158:
---
Author: ASF GitHub Bot
Created on: 08/Sep/20 09:11
Start Date: 08/Sep/20 09:11
Worklog Time Spent: 10m
Work Description: steveloughran merged pull request #2272:
URL: https://github.com/apache/hadoop/pull/2272
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479958)
Time Spent: 50m (was: 40m)
> Test timeout for ITestAbfsInputStreamStatistics#testReadAheadCounters
> -
>
> Key: HADOOP-17158
> URL: https://issues.apache.org/jira/browse/HADOOP-17158
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Intermittent test timeout for
> ITestAbfsInputStreamStatistics#testReadAheadCounters happening due to race
> conditions in readAhead threads.
> Test error:
> {code:java}
> [ERROR]
> testReadAheadCounters(org.apache.hadoop.fs.azurebfs.ITestAbfsInputStreamStatistics)
> Time elapsed: 30.723 s <<<
> ERROR!org.junit.runners.model.TestTimedOutException: test timed out after
> 3 milliseconds at java.lang.Thread.sleep(Native Method) at
> org.apache.hadoop.fs.azurebfs.ITestAbfsInputStreamStatistics.testReadAheadCounters(ITestAbfsInputStreamStatistics.java:346)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748) {code}
> Possible Reasoning:
> - ReadAhead queue doesn't get completed and hence the counter values are not
> satisfied in 30 seconds time for some systems.
> - The condition that readAheadBytesRead and remoteBytesRead counter values
> need to be greater than or equal to 4KB and 32KB respectively doesn't occur
> in some machines due to the fact that sometimes instead of reading for
> readAhead Buffer, remote reads are performed due to Threads still being in
> the readAhead queue to fill that buffer. Thus resulting in either of the 2
> counter values to be not satisfying the condition and getting in an infinite
> loop and hence timing out the test eventually.
> Possible Fixes:
> - Write better test(That would pass under all conditions).
> - Maybe UT instead of IT?
> Possible fix to better the test would be preferable and UT as the last resort.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17158) Test timeout for ITestAbfsInputStreamStatistics#testReadAheadCounters
[
https://issues.apache.org/jira/browse/HADOOP-17158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-17158.
-
Fix Version/s: 3.4.0
Resolution: Fixed
fixed in trunk; happy to cherrypick to -3.3 if you are
> Test timeout for ITestAbfsInputStreamStatistics#testReadAheadCounters
> -
>
> Key: HADOOP-17158
> URL: https://issues.apache.org/jira/browse/HADOOP-17158
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Intermittent test timeout for
> ITestAbfsInputStreamStatistics#testReadAheadCounters happening due to race
> conditions in readAhead threads.
> Test error:
> {code:java}
> [ERROR]
> testReadAheadCounters(org.apache.hadoop.fs.azurebfs.ITestAbfsInputStreamStatistics)
> Time elapsed: 30.723 s <<<
> ERROR!org.junit.runners.model.TestTimedOutException: test timed out after
> 3 milliseconds at java.lang.Thread.sleep(Native Method) at
> org.apache.hadoop.fs.azurebfs.ITestAbfsInputStreamStatistics.testReadAheadCounters(ITestAbfsInputStreamStatistics.java:346)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748) {code}
> Possible Reasoning:
> - ReadAhead queue doesn't get completed and hence the counter values are not
> satisfied in 30 seconds time for some systems.
> - The condition that readAheadBytesRead and remoteBytesRead counter values
> need to be greater than or equal to 4KB and 32KB respectively doesn't occur
> in some machines due to the fact that sometimes instead of reading for
> readAhead Buffer, remote reads are performed due to Threads still being in
> the readAhead queue to fill that buffer. Thus resulting in either of the 2
> counter values to be not satisfying the condition and getting in an infinite
> loop and hence timing out the test eventually.
> Possible Fixes:
> - Write better test(That would pass under all conditions).
> - Maybe UT instead of IT?
> Possible fix to better the test would be preferable and UT as the last resort.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #2272: HADOOP-17158. Test timeout for ITestAbfsInputStreamStatistics#testReadAheadCounters
steveloughran merged pull request #2272: URL: https://github.com/apache/hadoop/pull/2272 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17158) Test timeout for ITestAbfsInputStreamStatistics#testReadAheadCounters
[
https://issues.apache.org/jira/browse/HADOOP-17158?focusedWorklogId=479959=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479959
]
ASF GitHub Bot logged work on HADOOP-17158:
---
Author: ASF GitHub Bot
Created on: 08/Sep/20 09:11
Start Date: 08/Sep/20 09:11
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2272:
URL: https://github.com/apache/hadoop/pull/2272#issuecomment-688734784
merged to trunk; if you want it in branch-3.3 do a cherrypick, retest and
let me know it worked
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479959)
Time Spent: 1h (was: 50m)
> Test timeout for ITestAbfsInputStreamStatistics#testReadAheadCounters
> -
>
> Key: HADOOP-17158
> URL: https://issues.apache.org/jira/browse/HADOOP-17158
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Intermittent test timeout for
> ITestAbfsInputStreamStatistics#testReadAheadCounters happening due to race
> conditions in readAhead threads.
> Test error:
> {code:java}
> [ERROR]
> testReadAheadCounters(org.apache.hadoop.fs.azurebfs.ITestAbfsInputStreamStatistics)
> Time elapsed: 30.723 s <<<
> ERROR!org.junit.runners.model.TestTimedOutException: test timed out after
> 3 milliseconds at java.lang.Thread.sleep(Native Method) at
> org.apache.hadoop.fs.azurebfs.ITestAbfsInputStreamStatistics.testReadAheadCounters(ITestAbfsInputStreamStatistics.java:346)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748) {code}
> Possible Reasoning:
> - ReadAhead queue doesn't get completed and hence the counter values are not
> satisfied in 30 seconds time for some systems.
> - The condition that readAheadBytesRead and remoteBytesRead counter values
> need to be greater than or equal to 4KB and 32KB respectively doesn't occur
> in some machines due to the fact that sometimes instead of reading for
> readAhead Buffer, remote reads are performed due to Threads still being in
> the readAhead queue to fill that buffer. Thus resulting in either of the 2
> counter values to be not satisfying the condition and getting in an infinite
> loop and hence timing out the test eventually.
> Possible Fixes:
> - Write better test(That would pass under all conditions).
> - Maybe UT instead of IT?
> Possible fix to better the test would be preferable and UT as the last resort.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2273: HADOOP-17242. S3A ObjectListingIterator to block in hasNext() for results
steveloughran commented on a change in pull request #2273:
URL: https://github.com/apache/hadoop/pull/2273#discussion_r484767016
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java
##
@@ -764,82 +759,122 @@ public int getBatchSize() {
S3ListRequest request) throws IOException {
this.listPath = listPath;
this.maxKeys = listingOperationCallbacks.getMaxKeys();
+ this.request = request;
+ // initiate the asynchronous listing
this.s3ListResultFuture = listingOperationCallbacks
.listObjectsAsync(request);
- this.request = request;
- this.objectsPrev = null;
}
/**
- * Declare that the iterator has data if it is either is the initial
- * iteration or it is a later one and the last listing obtained was
- * incomplete.
- * @throws IOException never: there is no IO in this operation.
+ * Determine if there is more data.
+ *
+ * If the previous fetch
+ * completed and has not had its data returned by a next()
+ * call, then return true.
+ *
+ * Otherwise, wait for any outsanding listing to return, blocking for
+ * the result, and kick off the next async fetch if that result
+ * indicates there is more data.
+ *
+ * @throws IOException if the retrieval process failed.
*/
@Override
public boolean hasNext() throws IOException {
- return firstListing ||
- (objectsPrev != null && objectsPrev.isTruncated());
+ // there is a next element if either there is a result already
+ // or there is a result to block for
+ try {
+if (objects != null) {
+ // already a result, which next() will return.
+ return true;
+}
+// no previous objects
+
+// Is there is any active fetch underway?
+if (s3ListResultFuture == null) {
+ // no outstanding list, so no more results.
+ return false;
+}
+// there is a future, so wait for it.
+objects = awaitListOperationCompletion();
+listingCount++;
+// kick off the next listing
+fetchNextBatchAsyncIfPresent();
+
+// if we get here, there was a result, even if it is an empty listing.
+return true;
+ } catch (AmazonClientException e) {
+throw translateException("listObjects()", listPath, e);
+ }
+}
+
+/**
+ * Wait for the listing to complete; sets {@link #s3ListResultFuture}
+ * to null before returning.
+ * @return the result of the active operation.
+ * @throws IOException failure.
+ */
+protected S3ListResult awaitListOperationCompletion()
+throws IOException {
+ Preconditions.checkState(s3ListResultFuture != null,
+ "No active listing in progress");
+ S3ListResult result = awaitFuture(s3ListResultFuture);
Review comment:
If I add a once() here I can cut the catch/translate elsewhere
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17242) S3A (async) ObjectListingIterator to block in hasNext() for results
[
https://issues.apache.org/jira/browse/HADOOP-17242?focusedWorklogId=479951=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479951
]
ASF GitHub Bot logged work on HADOOP-17242:
---
Author: ASF GitHub Bot
Created on: 08/Sep/20 09:07
Start Date: 08/Sep/20 09:07
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#2273:
URL: https://github.com/apache/hadoop/pull/2273#discussion_r484767016
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java
##
@@ -764,82 +759,122 @@ public int getBatchSize() {
S3ListRequest request) throws IOException {
this.listPath = listPath;
this.maxKeys = listingOperationCallbacks.getMaxKeys();
+ this.request = request;
+ // initiate the asynchronous listing
this.s3ListResultFuture = listingOperationCallbacks
.listObjectsAsync(request);
- this.request = request;
- this.objectsPrev = null;
}
/**
- * Declare that the iterator has data if it is either is the initial
- * iteration or it is a later one and the last listing obtained was
- * incomplete.
- * @throws IOException never: there is no IO in this operation.
+ * Determine if there is more data.
+ *
+ * If the previous fetch
+ * completed and has not had its data returned by a next()
+ * call, then return true.
+ *
+ * Otherwise, wait for any outsanding listing to return, blocking for
+ * the result, and kick off the next async fetch if that result
+ * indicates there is more data.
+ *
+ * @throws IOException if the retrieval process failed.
*/
@Override
public boolean hasNext() throws IOException {
- return firstListing ||
- (objectsPrev != null && objectsPrev.isTruncated());
+ // there is a next element if either there is a result already
+ // or there is a result to block for
+ try {
+if (objects != null) {
+ // already a result, which next() will return.
+ return true;
+}
+// no previous objects
+
+// Is there is any active fetch underway?
+if (s3ListResultFuture == null) {
+ // no outstanding list, so no more results.
+ return false;
+}
+// there is a future, so wait for it.
+objects = awaitListOperationCompletion();
+listingCount++;
+// kick off the next listing
+fetchNextBatchAsyncIfPresent();
+
+// if we get here, there was a result, even if it is an empty listing.
+return true;
+ } catch (AmazonClientException e) {
+throw translateException("listObjects()", listPath, e);
+ }
+}
+
+/**
+ * Wait for the listing to complete; sets {@link #s3ListResultFuture}
+ * to null before returning.
+ * @return the result of the active operation.
+ * @throws IOException failure.
+ */
+protected S3ListResult awaitListOperationCompletion()
+throws IOException {
+ Preconditions.checkState(s3ListResultFuture != null,
+ "No active listing in progress");
+ S3ListResult result = awaitFuture(s3ListResultFuture);
Review comment:
If I add a once() here I can cut the catch/translate elsewhere
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479951)
Time Spent: 1h (was: 50m)
> S3A (async) ObjectListingIterator to block in hasNext() for results
> ---
>
> Key: HADOOP-17242
> URL: https://issues.apache.org/jira/browse/HADOOP-17242
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> HADOOP-17074 made listing async in S3A, but the iterator's hasNext Call
> doesn't wait for the result. If invoked on an empty path it *may* return when
> it should be failing.
> Note: surfaced in code review, not seen in the wild and all our tests were
> happy
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17229) Test failure as failed request body counted in byte received metric - ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
[
https://issues.apache.org/jira/browse/HADOOP-17229?focusedWorklogId=479961=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479961
]
ASF GitHub Bot logged work on HADOOP-17229:
---
Author: ASF GitHub Bot
Created on: 08/Sep/20 09:14
Start Date: 08/Sep/20 09:14
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2264:
URL: https://github.com/apache/hadoop/pull/2264#issuecomment-688736659
+1, merged to trunk, do a retest on branch-3.3 and I'll merge there too
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479961)
Time Spent: 2h (was: 1h 50m)
> Test failure as failed request body counted in byte received metric -
> ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
> ---
>
> Key: HADOOP-17229
> URL: https://issues.apache.org/jira/browse/HADOOP-17229
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, test
>Affects Versions: 3.3.0
>Reporter: Sneha Vijayarajan
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: abfsactive, pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Bytes received counter increments for every request response received.
> [https://github.com/apache/hadoop/blob/d23cc9d85d887f01d72180bdf1af87dfdee15c5a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java#L251]
> This increments even for failed requests.
> Observed during testing done for HADOOP-17215. A request failed with 409
> Conflict contains response body as below:
> {"error":\{"code":"PathAlreadyExists","message":"The specified path already
> exists.\nRequestId:c3b2c55c-b01f-0061-7b31-7b6ee300\nTime:2020-08-25T22:44:07.2356054Z"}}
> The error body of 168 size is incremented in bytes_received counter.
> This also breaks the testcase testAbfsHttpResponseStatistics.
> {code:java}
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR] Tests run: 2,
> Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR]
> testAbfsHttpResponseStatistics(org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics)
> Time elapsed: 13.183 s <<< FAILURE!java.lang.AssertionError: Mismatch in
> bytes_received expected:<143> but was:<311> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.assertAbfsStatistics(AbstractAbfsIntegrationTest.java:445)
> at
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics.testAbfsHttpResponseStatistics(ITestAbfsNetworkStatistics.java:291)
> {code}
> [~mehakmeetSingh] - is the bytes_received counter increment for failed
> requests an expected behaviour ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
hadoop-yetus removed a comment on pull request #2257: URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688314591 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 33m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | -1 :x: | mvninstall | 0m 22s | root in trunk failed. | | -1 :x: | compile | 0m 11s | hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 0m 25s | hadoop-aws in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 0m 20s | The patch fails to run checkstyle in hadoop-aws | | -1 :x: | mvnsite | 0m 10s | hadoop-aws in trunk failed. | | -1 :x: | shadedclient | 0m 49s | branch has errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 16s | hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javadoc | 0m 22s | hadoop-aws in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | +0 :ok: | spotbugs | 1m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 :x: | findbugs | 0m 22s | hadoop-aws in trunk failed. | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 41s | hadoop-aws in the patch failed. | | -1 :x: | compile | 0m 23s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javac | 0m 23s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 0m 22s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | javac | 0m 22s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 0m 5s | The patch fails to run checkstyle in hadoop-aws | | -1 :x: | mvnsite | 0m 12s | hadoop-aws in the patch failed. | | -1 :x: | whitespace | 0m 0s | The patch has 38 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | -1 :x: | whitespace | 0m 0s | The patch 600 line(s) with tabs. | | +1 :green_heart: | shadedclient | 0m 21s | patch has no errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 22s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javadoc | 0m 23s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | findbugs | 0m 23s | hadoop-aws in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 22s | hadoop-aws in the patch failed. | | +0 :ok: | asflicense | 0m 22s | ASF License check generated no output? | | | | 43m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2257 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 667bf71456f4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 34fe74da0e9 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt | | compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt | | checkstyle |
[jira] [Work logged] (HADOOP-17023) Tune listStatus() api of s3a.
[ https://issues.apache.org/jira/browse/HADOOP-17023?focusedWorklogId=480001=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480001 ] ASF GitHub Bot logged work on HADOOP-17023: --- Author: ASF GitHub Bot Created on: 08/Sep/20 10:13 Start Date: 08/Sep/20 10:13 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2257: URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688314591 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 33m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | -1 :x: | mvninstall | 0m 22s | root in trunk failed. | | -1 :x: | compile | 0m 11s | hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 0m 25s | hadoop-aws in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 0m 20s | The patch fails to run checkstyle in hadoop-aws | | -1 :x: | mvnsite | 0m 10s | hadoop-aws in trunk failed. | | -1 :x: | shadedclient | 0m 49s | branch has errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 16s | hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javadoc | 0m 22s | hadoop-aws in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | +0 :ok: | spotbugs | 1m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 :x: | findbugs | 0m 22s | hadoop-aws in trunk failed. | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 41s | hadoop-aws in the patch failed. | | -1 :x: | compile | 0m 23s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javac | 0m 23s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 0m 22s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | javac | 0m 22s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 0m 5s | The patch fails to run checkstyle in hadoop-aws | | -1 :x: | mvnsite | 0m 12s | hadoop-aws in the patch failed. | | -1 :x: | whitespace | 0m 0s | The patch has 38 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | -1 :x: | whitespace | 0m 0s | The patch 600 line(s) with tabs. | | +1 :green_heart: | shadedclient | 0m 21s | patch has no errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 22s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javadoc | 0m 23s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | findbugs | 0m 23s | hadoop-aws in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 22s | hadoop-aws in the patch failed. | | +0 :ok: | asflicense | 0m 22s | ASF License check generated no output? | | | | 43m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2257 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 667bf71456f4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 34fe74da0e9 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-mvninstall-root.txt | | compile |
[GitHub] [hadoop] lamber-ken opened a new pull request #2284: HDFS-15561. Fix NullPointException when start dfsrouter
lamber-ken opened a new pull request #2284: URL: https://github.com/apache/hadoop/pull/2284 ## NOTICE `dfs.federation.router.monitor.localnamenode.enable` is true by default, but it always add it to namenode heartbeat service, so the log file is full of exception information. ``` java.lang.IllegalArgumentException: java.net.UnknownHostException: null at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:447) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:171) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:123) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:95) at org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.getNamenodeStatusReport(NamenodeHeartbeatService.java:248) at org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.updateState(NamenodeHeartbeatService.java:205) at org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.periodicInvoke(NamenodeHeartbeatService.java:159) at org.apache.hadoop.hdfs.server.federation.router.PeriodicService$1.run(PeriodicService.java:178) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:300) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.base/java.lang.Thread.run(Thread.java:844) Caused by: java.net.UnknownHostException: null ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2283: HADOOP-17229. No updation of bytes received counter value after response failure occurs in ABFS
steveloughran commented on pull request #2283: URL: https://github.com/apache/hadoop/pull/2283#issuecomment-688833532 merged; closing. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #2283: HADOOP-17229. No updation of bytes received counter value after response failure occurs in ABFS
steveloughran closed pull request #2283: URL: https://github.com/apache/hadoop/pull/2283 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17229) Test failure as failed request body counted in byte received metric - ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
[
https://issues.apache.org/jira/browse/HADOOP-17229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-17229:
Fix Version/s: (was: 3.4.0)
3.3.1
> Test failure as failed request body counted in byte received metric -
> ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
> ---
>
> Key: HADOOP-17229
> URL: https://issues.apache.org/jira/browse/HADOOP-17229
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, test
>Affects Versions: 3.3.0
>Reporter: Sneha Vijayarajan
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: abfsactive, pull-request-available
> Fix For: 3.3.1
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Bytes received counter increments for every request response received.
> [https://github.com/apache/hadoop/blob/d23cc9d85d887f01d72180bdf1af87dfdee15c5a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java#L251]
> This increments even for failed requests.
> Observed during testing done for HADOOP-17215. A request failed with 409
> Conflict contains response body as below:
> {"error":\{"code":"PathAlreadyExists","message":"The specified path already
> exists.\nRequestId:c3b2c55c-b01f-0061-7b31-7b6ee300\nTime:2020-08-25T22:44:07.2356054Z"}}
> The error body of 168 size is incremented in bytes_received counter.
> This also breaks the testcase testAbfsHttpResponseStatistics.
> {code:java}
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR] Tests run: 2,
> Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR]
> testAbfsHttpResponseStatistics(org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics)
> Time elapsed: 13.183 s <<< FAILURE!java.lang.AssertionError: Mismatch in
> bytes_received expected:<143> but was:<311> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.assertAbfsStatistics(AbstractAbfsIntegrationTest.java:445)
> at
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics.testAbfsHttpResponseStatistics(ITestAbfsNetworkStatistics.java:291)
> {code}
> [~mehakmeetSingh] - is the bytes_received counter increment for failed
> requests an expected behaviour ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16918) Dependency update for Hadoop 2.10
[ https://issues.apache.org/jira/browse/HADOOP-16918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192185#comment-17192185 ] Masatake Iwasaki commented on HADOOP-16918: --- nimbus-jose-jwt was upgraded from 4.41.1 to 7.9 by dependabot. The commit is [c1d393a1567cac1bcf71e2e5f252cddffa0f97cc|https://github.com/apache/hadoop/commit/c1d393a1567cac1bcf71e2e5f252cddffa0f97cc]. I cherry-picked this to branch-2.10. This is 1 line fix of hadoop-project.xml and tests of hadoop-auth passed on my local. > Dependency update for Hadoop 2.10 > - > > Key: HADOOP-16918 > URL: https://issues.apache.org/jira/browse/HADOOP-16918 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Priority: Major > Labels: release-blocker > Attachments: dependency-check-report.html > > > A number of dependencies can be updated. > nimbus-jose-jwt > jetty > netty > zookeeper > hbase-common > jackson-databind > and many more. They should be updated in the 2.10.1 release. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet opened a new pull request #2282: HADOOP-17113, Hadoop-17158. Adding ReadAhead Counters and fixing ITestAbfsInputStreamStatistics#testReadAheadCounters
mehakmeet opened a new pull request #2282: URL: https://github.com/apache/hadoop/pull/2282 Backport for both HADOOP-17113(#2154) and HADOOP-17158(#2272) Tested by: mvn -T 1C -Dparallel-tests=abfs clean verify Region: East US ``` [INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 ``` ``` [INFO] Results: [INFO] [WARNING] Tests run: 451, Failures: 0, Errors: 0, Skipped: 71 ``` ``` [INFO] Results: [INFO] [WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 29 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17244) HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
[
https://issues.apache.org/jira/browse/HADOOP-17244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192171#comment-17192171
]
Steve Loughran commented on HADOOP-17244:
-
This is only an issue on branches with HADOOP-16430
> HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
> --
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
mukund-thakur commented on pull request #2257: URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688823862 > checkstyle complaining with fixable issues > > ``` > ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java:370: Listing.FileStatusListingIterator filesItr = createFileStatusListingIterator(: Line is longer than 80 characters (found 81). [LineLength] > ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:34:import java.util.Arrays;:8: Unused import - java.util.Arrays. [UnusedImports] > ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2672: private RemoteIterator innerListStatus(Path f) throws FileNotFoundException,: Line is longer than 80 characters (found 93). [LineLength] > ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2686: }:5: '}' at column 5 should be on the same line as the next part of a multi-block statement (one that directly contains multiple blocks: if/else-if/else, do/while or try/catch/finally). [RightCurly] > ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java:45:import org.apache.hadoop.fs.s3a.impl.TestPartialDeleteFailures;:8: Unused import - org.apache.hadoop.fs.s3a.impl.TestPartialDeleteFailures. [UnusedImports] > ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java:138: S3AFileStatus[] result = S3AUtils.iteratorToStatuses(resultItr, new HashSet<>());: Line is longer than 80 characters (found 85). [LineLength] > ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java:205: S3AFileStatus[] result = S3AUtils.iteratorToStatuses(resultItr, new HashSet<>());: Line is longer than 80 characters (found 85). [LineLength] > ``` Yes I am going to fix the checkstyle. How is the rest of code? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17023) Tune listStatus() api of s3a.
[ https://issues.apache.org/jira/browse/HADOOP-17023?focusedWorklogId=480084=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480084 ] ASF GitHub Bot logged work on HADOOP-17023: --- Author: ASF GitHub Bot Created on: 08/Sep/20 12:08 Start Date: 08/Sep/20 12:08 Worklog Time Spent: 10m Work Description: mukund-thakur commented on pull request #2257: URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688823862 > checkstyle complaining with fixable issues > > ``` > ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java:370: Listing.FileStatusListingIterator filesItr = createFileStatusListingIterator(: Line is longer than 80 characters (found 81). [LineLength] > ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:34:import java.util.Arrays;:8: Unused import - java.util.Arrays. [UnusedImports] > ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2672: private RemoteIterator innerListStatus(Path f) throws FileNotFoundException,: Line is longer than 80 characters (found 93). [LineLength] > ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2686: }:5: '}' at column 5 should be on the same line as the next part of a multi-block statement (one that directly contains multiple blocks: if/else-if/else, do/while or try/catch/finally). [RightCurly] > ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java:45:import org.apache.hadoop.fs.s3a.impl.TestPartialDeleteFailures;:8: Unused import - org.apache.hadoop.fs.s3a.impl.TestPartialDeleteFailures. [UnusedImports] > ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java:138: S3AFileStatus[] result = S3AUtils.iteratorToStatuses(resultItr, new HashSet<>());: Line is longer than 80 characters (found 85). [LineLength] > ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java:205: S3AFileStatus[] result = S3AUtils.iteratorToStatuses(resultItr, new HashSet<>());: Line is longer than 80 characters (found 85). [LineLength] > ``` Yes I am going to fix the checkstyle. How is the rest of code? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 480084) Time Spent: 1h 50m (was: 1h 40m) > Tune listStatus() api of s3a. > - > > Key: HADOOP-17023 > URL: https://issues.apache.org/jira/browse/HADOOP-17023 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.1 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > Similar optimisation which was done for listLocatedSttaus() > https://issues.apache.org/jira/browse/HADOOP-16465 can done for listStatus() > api as well. > This is going to reduce the number of remote calls in case of directory > listing. > > CC [~ste...@apache.org] [~shwethags] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
steveloughran commented on a change in pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#discussion_r484868210
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java
##
@@ -362,14 +340,16 @@ public static BulkOperationState initiateBulkWrite(
* @return Final result of directory listing.
* @throws IOException if metadata store update failed
*/
- public static S3AFileStatus[] dirListingUnion(MetadataStore ms, Path path,
- List backingStatuses, DirListingMetadata dirMeta,
- boolean isAuthoritative, ITtlTimeProvider timeProvider)
- throws IOException {
+ public static RemoteIterator dirListingUnion(
+ MetadataStore ms, Path path,
+ RemoteIterator backingStatuses,
+ DirListingMetadata dirMeta, boolean isAuthoritative,
+ ITtlTimeProvider timeProvider, Listing listing)
Review comment:
we are into callback country here. Could you
1. create an interface (say in S3Guard) for
createProvidedFileStatusIteratorForS3Guard (it could contain all params for
`createProvidedFileStatusIterator` oterh than the dir meta list, 2. Have
Listing implement it
3. Pass the interface in, not the reference to Listing.
I'm just trying to move away from having all the classes in here having
intimate knowledge of every other class. An interface with a single method lets
us control exactly what S3Guard does.
You could use a Lambda-expression if you want -for a single operation it
would work, just take `Function`
as a parameter and pass the operation in that way. But, given you use this
operation in tests as well as production code, you should still implement this
function as a single method in Listing
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17023) Tune listStatus() api of s3a.
[
https://issues.apache.org/jira/browse/HADOOP-17023?focusedWorklogId=480090=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480090
]
ASF GitHub Bot logged work on HADOOP-17023:
---
Author: ASF GitHub Bot
Created on: 08/Sep/20 12:20
Start Date: 08/Sep/20 12:20
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#2257:
URL: https://github.com/apache/hadoop/pull/2257#discussion_r484868210
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java
##
@@ -362,14 +340,16 @@ public static BulkOperationState initiateBulkWrite(
* @return Final result of directory listing.
* @throws IOException if metadata store update failed
*/
- public static S3AFileStatus[] dirListingUnion(MetadataStore ms, Path path,
- List backingStatuses, DirListingMetadata dirMeta,
- boolean isAuthoritative, ITtlTimeProvider timeProvider)
- throws IOException {
+ public static RemoteIterator dirListingUnion(
+ MetadataStore ms, Path path,
+ RemoteIterator backingStatuses,
+ DirListingMetadata dirMeta, boolean isAuthoritative,
+ ITtlTimeProvider timeProvider, Listing listing)
Review comment:
we are into callback country here. Could you
1. create an interface (say in S3Guard) for
createProvidedFileStatusIteratorForS3Guard (it could contain all params for
`createProvidedFileStatusIterator` oterh than the dir meta list, 2. Have
Listing implement it
3. Pass the interface in, not the reference to Listing.
I'm just trying to move away from having all the classes in here having
intimate knowledge of every other class. An interface with a single method lets
us control exactly what S3Guard does.
You could use a Lambda-expression if you want -for a single operation it
would work, just take `Function`
as a parameter and pass the operation in that way. But, given you use this
operation in tests as well as production code, you should still implement this
function as a single method in Listing
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 480090)
Time Spent: 2h (was: 1h 50m)
> Tune listStatus() api of s3a.
> -
>
> Key: HADOOP-17023
> URL: https://issues.apache.org/jira/browse/HADOOP-17023
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.1
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Similar optimisation which was done for listLocatedSttaus()
> https://issues.apache.org/jira/browse/HADOOP-16465 can done for listStatus()
> api as well.
> This is going to reduce the number of remote calls in case of directory
> listing.
>
> CC [~ste...@apache.org] [~shwethags]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17244) HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
[
https://issues.apache.org/jira/browse/HADOOP-17244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192080#comment-17192080
]
Steve Loughran commented on HADOOP-17244:
-
.. this fix needs to go in to all releases of Hadoop which do incremental
s3guard updates during delete. I think that is 3.3+ only.
> HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
> --
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17023) Tune listStatus() api of s3a.
[ https://issues.apache.org/jira/browse/HADOOP-17023?focusedWorklogId=479991=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479991 ] ASF GitHub Bot logged work on HADOOP-17023: --- Author: ASF GitHub Bot Created on: 08/Sep/20 10:02 Start Date: 08/Sep/20 10:02 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2257: URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688762761 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 31m 52s | trunk passed | | +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 27s | trunk passed | | +1 :green_heart: | mvnsite | 0m 38s | trunk passed | | +1 :green_heart: | shadedclient | 17m 20s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 19s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 27s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 0s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 33s | the patch passed | | +1 :green_heart: | compile | 0m 33s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 33s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 28s | the patch passed | | -0 :warning: | checkstyle | 0m 20s | hadoop-tools/hadoop-aws: The patch generated 7 new + 62 unchanged - 0 fixed = 69 total (was 62) | | +1 :green_heart: | mvnsite | 0m 31s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 16m 6s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 17s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 39s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 78m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2257 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 27053c1e385f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / ac7d4623aef | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/testReport/ | | Max. process+thread count | 360 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/console | | versions |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
hadoop-yetus commented on pull request #2257: URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688762761 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 31m 52s | trunk passed | | +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 27s | trunk passed | | +1 :green_heart: | mvnsite | 0m 38s | trunk passed | | +1 :green_heart: | shadedclient | 17m 20s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 19s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 27s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 0s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 33s | the patch passed | | +1 :green_heart: | compile | 0m 33s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 33s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 28s | the patch passed | | -0 :warning: | checkstyle | 0m 20s | hadoop-tools/hadoop-aws: The patch generated 7 new + 62 unchanged - 0 fixed = 69 total (was 62) | | +1 :green_heart: | mvnsite | 0m 31s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 16m 6s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 17s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 39s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 78m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2257 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 27053c1e385f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / ac7d4623aef | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/testReport/ | | Max. process+thread count | 360 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/console | | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 | | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this
[jira] [Commented] (HADOOP-17217) S3A FileSystem does not correctly delete directories with fake entries
[
https://issues.apache.org/jira/browse/HADOOP-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192123#comment-17192123
]
Steve Loughran commented on HADOOP-17217:
-
Should be fixed in HADOOP-13230, provided you don't turn S3Guard on
[HADOOP-17244].
[~kupferk] -can you check out and build apache hadoop trunk (or branch-3.3) and
test it against your store there? If not, then that JIRA is incomplete.
> S3A FileSystem does not correctly delete directories with fake entries
> --
>
> Key: HADOOP-17217
> URL: https://issues.apache.org/jira/browse/HADOOP-17217
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Kaya Kupferschmidt
>Priority: Major
>
> h3. Summary
> We are facing an issue where the Hadoop S3A Filesystem gets confused by fake
> directory objects in S3. Specifically trying to recursively remove a whole
> Hadoop directory (i.e. all objects with the same S3 prefix) doesn't work as
> expected.
> h2. Background
> We are using Alluxio together with S3 as our deep store. For some
> infrastructure reasons we decided to directly write to S3 (bypassing Alluxio)
> with our Spark applications, and only use Alluxio for reading data.
> When we directly write our results into S3 with Spark, everything is fine.
> But once Alluxio accesses S3, it will create these fake directory entries.
> When we now try to overwrite existing data in S3 with a Spark application,
> the result is incorrect, since Spark will only remove the fake directory
> entry, but not all other objects below that prefix in S3.
> Of course it is questionable if Alluxio is doing the right thing, but on the
> other hand it seems that Hadoop also does not behave as we expected.
> h2. Steps to Reproduce
> The following steps only require AWS CLI and Hadoop CLI to reproduce the
> issue we are facing:
> h3. Initial setup
> {code:bash}
> # First step: Create an new and empty bucket in S3
> $ aws s3 mb s3://dimajix-tmp
>
> make_bucket: dimajix-tmp
> $ aws s3 ls
> 2020-08-21 11:19:50 dimajix-tmp
> # Upload some data
> $ aws s3 cp some_file.txt s3://dimajix-tmp/tmp/
> upload: ./some_file.txt to s3://dimajix-tmp/tmp/some_file.txt
> $ aws s3 ls s3://dimajix-tmp/tmp/
>
> 2020-08-21 11:23:35 0 some_file.txt
> # Check that Hadoop can list the file
> $ /opt/hadoop/bin/hdfs dfs -ls s3a://dimajix-tmp/
> Found 1 items
> drwxrwxrwx - kaya kaya 0 2020-08-21 11:24 s3a://dimajix-tmp/tmp
> $ /opt/hadoop/bin/hdfs dfs -ls s3a://dimajix-tmp/tmp/
> -rw-rw-rw- 1 kaya kaya 0 2020-08-21 11:23
> s3a://dimajix-tmp/tmp/some_file.txt
> # Evil step: Create fake directory entry in S3
> $ aws s3api put-object --bucket dimajix-tmp --key tmp/
> {
> "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\""
> }
> # Look into S3, ensure that fake directory entry was created
> $ aws s3 ls s3://dimajix-tmp/tmp/
> 2020-08-21 11:25:40 0
> 2020-08-21 11:23:35 0 some_file.txt
> # Look into S3 using Hadoop CLI, ensure that everything looks okay (which is
> the case)
> $ /opt/hadoop/bin/hdfs dfs -ls s3a://dimajix-tmp/tmp/
> Found 1 items
> -rw-rw-rw- 1 kaya kaya 0 2020-08-21 11:23
> s3a://dimajix-tmp/tmp/some_file.txt
> {code}
> h3. Reproduce questionable behaviour: Try to recursively delete directory
> {code:bash}
> # Bug: Now try to delete the directory with Hadoop CLI
> $ /opt/hadoop/bin/hdfs dfs -rm s3a://dimajix-tmp/tmp/
> rm: `s3a://dimajix-tmp/tmp': Is a directory
> # Okay, that didn't work out, Hadoop interprets the prefix as a directory
> (which is fine). It also did not delete anything, as we can see in S3:
> $ aws s3 ls s3://dimajix-tmp/tmp/
> 2020-08-21 11:25:40 0
> 2020-08-21 11:23:35 0 some_file.txt
> # Now let's try a little bit harder by trying to recursively delete the
> directory
> $ /opt/hadoop/bin/hdfs dfs -rm -r s3a://dimajix-tmp/tmp/
> Deleted s3a://dimajix-tmp/tmp
> # Everything looked fine so far. But let's inspect S3 directly. We'll find
> that only the prefix (fake directory entry) has been removed. The file in the
> directory is still there.
> $ aws s3 ls s3://dimajix-tmp/tmp/
> 2020-08-21 11:23:35 0 some_file.txt
> # We can also use Hadoop CLI to check that the directory containing the file
> is still present, although we wanted to delete it above.
> $ /opt/hadoop/bin/hdfs dfs -ls s3a://dimajix-tmp/tmp/
> Found 1 items
> -rw-rw-rw- 1 kaya kaya 0 2020-08-21 11:23
>
[jira] [Work logged] (HADOOP-17229) Test failure as failed request body counted in byte received metric - ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
[
https://issues.apache.org/jira/browse/HADOOP-17229?focusedWorklogId=480053=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480053
]
ASF GitHub Bot logged work on HADOOP-17229:
---
Author: ASF GitHub Bot
Created on: 08/Sep/20 11:18
Start Date: 08/Sep/20 11:18
Worklog Time Spent: 10m
Work Description: mehakmeet opened a new pull request #2283:
URL: https://github.com/apache/hadoop/pull/2283
Backport for HADOOP-17229(#2264)
Tested by: mvn -T 1C -Dparallel-tests=abfs clean verify
```
[INFO] Results:
[INFO]
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
```
```
[INFO] Results:
[INFO]
[WARNING] Tests run: 451, Failures: 0, Errors: 0, Skipped: 71
```
```
[INFO] Results:
[INFO]
[WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 29
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 480053)
Time Spent: 2h 10m (was: 2h)
> Test failure as failed request body counted in byte received metric -
> ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
> ---
>
> Key: HADOOP-17229
> URL: https://issues.apache.org/jira/browse/HADOOP-17229
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, test
>Affects Versions: 3.3.0
>Reporter: Sneha Vijayarajan
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: abfsactive, pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Bytes received counter increments for every request response received.
> [https://github.com/apache/hadoop/blob/d23cc9d85d887f01d72180bdf1af87dfdee15c5a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java#L251]
> This increments even for failed requests.
> Observed during testing done for HADOOP-17215. A request failed with 409
> Conflict contains response body as below:
> {"error":\{"code":"PathAlreadyExists","message":"The specified path already
> exists.\nRequestId:c3b2c55c-b01f-0061-7b31-7b6ee300\nTime:2020-08-25T22:44:07.2356054Z"}}
> The error body of 168 size is incremented in bytes_received counter.
> This also breaks the testcase testAbfsHttpResponseStatistics.
> {code:java}
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR] Tests run: 2,
> Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR]
> testAbfsHttpResponseStatistics(org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics)
> Time elapsed: 13.183 s <<< FAILURE!java.lang.AssertionError: Mismatch in
> bytes_received expected:<143> but was:<311> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.assertAbfsStatistics(AbstractAbfsIntegrationTest.java:445)
> at
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics.testAbfsHttpResponseStatistics(ITestAbfsNetworkStatistics.java:291)
> {code}
> [~mehakmeetSingh] - is the bytes_received counter increment for failed
> requests an expected behaviour ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] BilwaST removed a comment on pull request #2239: YARN-10403. Add success/failure message to RMAdmin CLI queue refresh
BilwaST removed a comment on pull request #2239: URL: https://github.com/apache/hadoop/pull/2239#issuecomment-688719152 Thanks @brucesw for updating. +1(Non binding) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17229) Test failure as failed request body counted in byte received metric - ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
[
https://issues.apache.org/jira/browse/HADOOP-17229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192179#comment-17192179
]
Steve Loughran commented on HADOOP-17229:
-
merged to trunk and 3.3
> Test failure as failed request body counted in byte received metric -
> ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
> ---
>
> Key: HADOOP-17229
> URL: https://issues.apache.org/jira/browse/HADOOP-17229
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, test
>Affects Versions: 3.3.0
>Reporter: Sneha Vijayarajan
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: abfsactive, pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Bytes received counter increments for every request response received.
> [https://github.com/apache/hadoop/blob/d23cc9d85d887f01d72180bdf1af87dfdee15c5a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java#L251]
> This increments even for failed requests.
> Observed during testing done for HADOOP-17215. A request failed with 409
> Conflict contains response body as below:
> {"error":\{"code":"PathAlreadyExists","message":"The specified path already
> exists.\nRequestId:c3b2c55c-b01f-0061-7b31-7b6ee300\nTime:2020-08-25T22:44:07.2356054Z"}}
> The error body of 168 size is incremented in bytes_received counter.
> This also breaks the testcase testAbfsHttpResponseStatistics.
> {code:java}
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR] Tests run: 2,
> Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR]
> testAbfsHttpResponseStatistics(org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics)
> Time elapsed: 13.183 s <<< FAILURE!java.lang.AssertionError: Mismatch in
> bytes_received expected:<143> but was:<311> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.assertAbfsStatistics(AbstractAbfsIntegrationTest.java:445)
> at
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics.testAbfsHttpResponseStatistics(ITestAbfsNetworkStatistics.java:291)
> {code}
> [~mehakmeetSingh] - is the bytes_received counter increment for failed
> requests an expected behaviour ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17229) Test failure as failed request body counted in byte received metric - ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
[
https://issues.apache.org/jira/browse/HADOOP-17229?focusedWorklogId=480093=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480093
]
ASF GitHub Bot logged work on HADOOP-17229:
---
Author: ASF GitHub Bot
Created on: 08/Sep/20 12:28
Start Date: 08/Sep/20 12:28
Worklog Time Spent: 10m
Work Description: steveloughran closed pull request #2283:
URL: https://github.com/apache/hadoop/pull/2283
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 480093)
Time Spent: 2.5h (was: 2h 20m)
> Test failure as failed request body counted in byte received metric -
> ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
> ---
>
> Key: HADOOP-17229
> URL: https://issues.apache.org/jira/browse/HADOOP-17229
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, test
>Affects Versions: 3.3.0
>Reporter: Sneha Vijayarajan
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: abfsactive, pull-request-available
> Fix For: 3.3.1
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Bytes received counter increments for every request response received.
> [https://github.com/apache/hadoop/blob/d23cc9d85d887f01d72180bdf1af87dfdee15c5a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java#L251]
> This increments even for failed requests.
> Observed during testing done for HADOOP-17215. A request failed with 409
> Conflict contains response body as below:
> {"error":\{"code":"PathAlreadyExists","message":"The specified path already
> exists.\nRequestId:c3b2c55c-b01f-0061-7b31-7b6ee300\nTime:2020-08-25T22:44:07.2356054Z"}}
> The error body of 168 size is incremented in bytes_received counter.
> This also breaks the testcase testAbfsHttpResponseStatistics.
> {code:java}
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR] Tests run: 2,
> Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR]
> testAbfsHttpResponseStatistics(org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics)
> Time elapsed: 13.183 s <<< FAILURE!java.lang.AssertionError: Mismatch in
> bytes_received expected:<143> but was:<311> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.assertAbfsStatistics(AbstractAbfsIntegrationTest.java:445)
> at
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics.testAbfsHttpResponseStatistics(ITestAbfsNetworkStatistics.java:291)
> {code}
> [~mehakmeetSingh] - is the bytes_received counter increment for failed
> requests an expected behaviour ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17229) Test failure as failed request body counted in byte received metric - ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
[
https://issues.apache.org/jira/browse/HADOOP-17229?focusedWorklogId=480092=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480092
]
ASF GitHub Bot logged work on HADOOP-17229:
---
Author: ASF GitHub Bot
Created on: 08/Sep/20 12:28
Start Date: 08/Sep/20 12:28
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2283:
URL: https://github.com/apache/hadoop/pull/2283#issuecomment-688833532
merged; closing. Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 480092)
Time Spent: 2h 20m (was: 2h 10m)
> Test failure as failed request body counted in byte received metric -
> ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
> ---
>
> Key: HADOOP-17229
> URL: https://issues.apache.org/jira/browse/HADOOP-17229
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, test
>Affects Versions: 3.3.0
>Reporter: Sneha Vijayarajan
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: abfsactive, pull-request-available
> Fix For: 3.3.1
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Bytes received counter increments for every request response received.
> [https://github.com/apache/hadoop/blob/d23cc9d85d887f01d72180bdf1af87dfdee15c5a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java#L251]
> This increments even for failed requests.
> Observed during testing done for HADOOP-17215. A request failed with 409
> Conflict contains response body as below:
> {"error":\{"code":"PathAlreadyExists","message":"The specified path already
> exists.\nRequestId:c3b2c55c-b01f-0061-7b31-7b6ee300\nTime:2020-08-25T22:44:07.2356054Z"}}
> The error body of 168 size is incremented in bytes_received counter.
> This also breaks the testcase testAbfsHttpResponseStatistics.
> {code:java}
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR] Tests run: 2,
> Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR]
> testAbfsHttpResponseStatistics(org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics)
> Time elapsed: 13.183 s <<< FAILURE!java.lang.AssertionError: Mismatch in
> bytes_received expected:<143> but was:<311> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.assertAbfsStatistics(AbstractAbfsIntegrationTest.java:445)
> at
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics.testAbfsHttpResponseStatistics(ITestAbfsNetworkStatistics.java:291)
> {code}
> [~mehakmeetSingh] - is the bytes_received counter increment for failed
> requests an expected behaviour ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2282: HADOOP-17113, HADOOP-17158. Adding ReadAhead Counters and fixing ITestAbfsInputStreamStatistics#testReadAheadCounters
steveloughran commented on pull request #2282: URL: https://github.com/apache/hadoop/pull/2282#issuecomment-688851051 merged to branch-3.3. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #2282: HADOOP-17113, HADOOP-17158. Adding ReadAhead Counters and fixing ITestAbfsInputStreamStatistics#testReadAheadCounters
steveloughran closed pull request #2282: URL: https://github.com/apache/hadoop/pull/2282 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16918) Dependency update for Hadoop 2.10
[
https://issues.apache.org/jira/browse/HADOOP-16918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192196#comment-17192196
]
Masatake Iwasaki commented on HADOOP-16918:
---
difference of netty version between branch-2.10 and trunk.
{noformat}
$ git diff branch-2.10 trunk -- hadoop-project/pom.xml | grep -A1 netty
+3.10.6.Final
+4.1.50.Final
+
--
io.netty
netty
-3.10.6.Final
+${netty3.version}
--
io.netty
netty-all
-4.0.23.Final
+${netty4.version}
--
...
{noformat}
netty4 was upgraded
* from 4.1.48.Final to 4.1.50.Final by YARN-10265
* from 4.1.45.Final to 4.1.48.Final by HADOOP-16982
* from 4.1.42.Final to 4.1.45.Final by HADOOP-16871
* from 4.0.52.Final to 4.1.42.Final by HADOOP-16643
* from 4.0.23.Final to 4.0.52.Final by HADOOP-14910
I'm cherry-picking HADOOP-14910.
> Dependency update for Hadoop 2.10
> -
>
> Key: HADOOP-16918
> URL: https://issues.apache.org/jira/browse/HADOOP-16918
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Priority: Major
> Labels: release-blocker
> Attachments: dependency-check-report.html
>
>
> A number of dependencies can be updated.
> nimbus-jose-jwt
> jetty
> netty
> zookeeper
> hbase-common
> jackson-databind
> and many more. They should be updated in the 2.10.1 release.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2281: HDFS-15516.Add info for create flags in NameNode audit logs.
hadoop-yetus commented on pull request #2281: URL: https://github.com/apache/hadoop/pull/2281#issuecomment-688789950 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 12s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 33m 2s | trunk passed | | +1 :green_heart: | compile | 1m 19s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 1m 11s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 50s | trunk passed | | +1 :green_heart: | mvnsite | 1m 18s | trunk passed | | +1 :green_heart: | shadedclient | 18m 50s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 6s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 26s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 3m 39s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 37s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 27s | the patch passed | | +1 :green_heart: | compile | 1m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 1m 28s | the patch passed | | +1 :green_heart: | compile | 1m 15s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 1m 15s | the patch passed | | -0 :warning: | checkstyle | 0m 49s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 184 unchanged - 1 fixed = 187 total (was 185) | | +1 :green_heart: | mvnsite | 1m 25s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 17m 7s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 50s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 18s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | findbugs | 3m 30s | hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | -1 :x: | unit | 111m 21s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | | | | 209m 11s | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-hdfs-project/hadoop-hdfs | | | org.apache.hadoop.hdfs.server.namenode.FSNamesystem$FSNamesystemAuditLogger.logAuditEvent(boolean, String, InetAddress, String, String, String, FileStatus, EnumSet, CallerContext, UserGroupInformation, DelegationTokenSecretManager) concatenates strings using + in a loop At FSNamesystem.java:String, FileStatus, EnumSet, CallerContext, UserGroupInformation, DelegationTokenSecretManager) concatenates strings using + in a loop At FSNamesystem.java:[line 8655] | | Failed junit tests | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor | | | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.namenode.TestFsck | | | hadoop.hdfs.server.namenode.ha.TestHAAppend | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2281/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2281 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b0f97e6ec64c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / ac7d4623aef | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions |
[GitHub] [hadoop] BilwaST commented on pull request #2239: YARN-10403. Add success/failure message to RMAdmin CLI queue refresh
BilwaST commented on pull request #2239:
URL: https://github.com/apache/hadoop/pull/2239#issuecomment-688807826
Hi @brucesw
RMAdminCLI#run method already is printing error msg like below . You can add
success message in run method itself which will be common for all commands.
What do you think?
`
catch (Exception e) {
exitCode = -1;
System.err.println(cmd.substring(1) + ": "
+ e.getLocalizedMessage());
}
`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2282: HADOOP-17113, HADOOP-17158. Adding ReadAhead Counters and fixing ITestAbfsInputStreamStatistics#testReadAheadCounters
hadoop-yetus commented on pull request #2282: URL: https://github.com/apache/hadoop/pull/2282#issuecomment-688828740 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 31m 4s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-3.3 Compile Tests _ | | +1 :green_heart: | mvninstall | 28m 33s | branch-3.3 passed | | +1 :green_heart: | compile | 0m 32s | branch-3.3 passed | | +1 :green_heart: | checkstyle | 0m 27s | branch-3.3 passed | | +1 :green_heart: | mvnsite | 0m 37s | branch-3.3 passed | | +1 :green_heart: | shadedclient | 15m 4s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 27s | branch-3.3 passed | | +0 :ok: | spotbugs | 0m 55s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 52s | branch-3.3 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 30s | the patch passed | | +1 :green_heart: | compile | 0m 24s | the patch passed | | +1 :green_heart: | javac | 0m 24s | the patch passed | | +1 :green_heart: | checkstyle | 0m 16s | the patch passed | | +1 :green_heart: | mvnsite | 0m 27s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 51s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | | +1 :green_heart: | findbugs | 0m 55s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 23s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 98m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2282/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2282 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 54bd4a666c1f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-3.3 / 5710005 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2282/1/testReport/ | | Max. process+thread count | 418 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2282/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17189) add way for s3a to recognise buckets with "." in name and switch to path access
[ https://issues.apache.org/jira/browse/HADOOP-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17189. - Fix Version/s: 3.4.0 Resolution: Won't Fix WONTFIXing this, as buckets with . in their name will soon be forbidden > add way for s3a to recognise buckets with "." in name and switch to path > access > --- > > Key: HADOOP-17189 > URL: https://issues.apache.org/jira/browse/HADOOP-17189 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > Fix For: 3.4.0 > > > # AWS has, historically, allowed buckets with '.' in their name (along with > other non-DNS valid chars) > # none of which work with virtual hostname S3 clients -you have to enable > path style access > # which we can't do on a per-bucket basis, as the logic there doesn't support > buckets with '.' in the name (think about it...) > # and we can't blindly say "use path access everywhere", because all buckets > created on/after 2020-10-01 won't work that way > dest.set(Math.max(dest.get(), sourceValue)); -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17113) Adding ReadAhead Counters in ABFS
[ https://issues.apache.org/jira/browse/HADOOP-17113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17113: Labels: pull-request-available (was: ) > Adding ReadAhead Counters in ABFS > - > > Key: HADOOP-17113 > URL: https://issues.apache.org/jira/browse/HADOOP-17113 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Mehakmeet Singh >Assignee: Mehakmeet Singh >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Adding ReadAheads Counters in ABFS to track the behavior of the ReadAhead > feature in ABFS. This would include 2 counters: > |READ_AHEAD_BYTES_READ|number of bytes read by readAhead| > |READ_AHEAD_REMOTE_BYTES_READ|number of bytes not used after readAhead was > used| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17113) Adding ReadAhead Counters in ABFS
[ https://issues.apache.org/jira/browse/HADOOP-17113?focusedWorklogId=480016=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480016 ] ASF GitHub Bot logged work on HADOOP-17113: --- Author: ASF GitHub Bot Created on: 08/Sep/20 10:39 Start Date: 08/Sep/20 10:39 Worklog Time Spent: 10m Work Description: mehakmeet opened a new pull request #2282: URL: https://github.com/apache/hadoop/pull/2282 Backport for both HADOOP-17113(#2154) and HADOOP-17158(#2272) Tested by: mvn -T 1C -Dparallel-tests=abfs clean verify Region: East US ``` [INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 ``` ``` [INFO] Results: [INFO] [WARNING] Tests run: 451, Failures: 0, Errors: 0, Skipped: 71 ``` ``` [INFO] Results: [INFO] [WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 29 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 480016) Remaining Estimate: 0h Time Spent: 10m > Adding ReadAhead Counters in ABFS > - > > Key: HADOOP-17113 > URL: https://issues.apache.org/jira/browse/HADOOP-17113 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Mehakmeet Singh >Assignee: Mehakmeet Singh >Priority: Major > Fix For: 3.4.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Adding ReadAheads Counters in ABFS to track the behavior of the ReadAhead > feature in ABFS. This would include 2 counters: > |READ_AHEAD_BYTES_READ|number of bytes read by readAhead| > |READ_AHEAD_REMOTE_BYTES_READ|number of bytes not used after readAhead was > used| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17250) ABFS: Allow random read sizes to be of buffer size
Sneha Vijayarajan created HADOOP-17250: -- Summary: ABFS: Allow random read sizes to be of buffer size Key: HADOOP-17250 URL: https://issues.apache.org/jira/browse/HADOOP-17250 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.1.4 Reporter: Sneha Vijayarajan Assignee: Sneha Vijayarajan ADLS Gen2/ABFS driver is optimized to read only the bytes that are requested for when the read pattern is random. It was observed in some spark jobs that though the reads are random, the next read doesn't skip by a lot and can be served by the earlier read if read was done in buffer size. As a result the job triggered a higher count of read calls and resulted in higher job runtime. When these jobs were run against Gen1 which always reads in buffer size , the jobs fared well. In this Jira we try to provide a control over config on random read to be of requested size or buffer size. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet opened a new pull request #2283: HADOOP-17229. No updation of bytes received counter value after response failure occurs in ABFS
mehakmeet opened a new pull request #2283: URL: https://github.com/apache/hadoop/pull/2283 Backport for HADOOP-17229(#2264) Tested by: mvn -T 1C -Dparallel-tests=abfs clean verify ``` [INFO] Results: [INFO] [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 ``` ``` [INFO] Results: [INFO] [WARNING] Tests run: 451, Failures: 0, Errors: 0, Skipped: 71 ``` ``` [INFO] Results: [INFO] [WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 29 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
hadoop-yetus removed a comment on pull request #2257: URL: https://github.com/apache/hadoop/pull/2257#issuecomment-682588174 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 37m 46s | trunk passed | | +1 :green_heart: | compile | 1m 11s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 43s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 32s | trunk passed | | +1 :green_heart: | mvnsite | 0m 58s | trunk passed | | +1 :green_heart: | shadedclient | 20m 36s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 32s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 28s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 48s | the patch passed | | +1 :green_heart: | compile | 0m 48s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 48s | the patch passed | | +1 :green_heart: | compile | 0m 37s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 37s | the patch passed | | -0 :warning: | checkstyle | 0m 23s | hadoop-tools/hadoop-aws: The patch generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13) | | +1 :green_heart: | mvnsite | 0m 47s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 18m 48s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 18s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 5s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 1m 12s | The patch does not generate ASF License warnings. | | | | 96m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2257 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 1af3e1f76ff6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / bfd99f31fd6 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/testReport/ | | Max. process+thread count | 344 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/console | | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 | | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about
[GitHub] [hadoop] steveloughran commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
steveloughran commented on pull request #2257: URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688822205 checkstyle complaining with fixable issues ``` ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java:370: Listing.FileStatusListingIterator filesItr = createFileStatusListingIterator(: Line is longer than 80 characters (found 81). [LineLength] ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:34:import java.util.Arrays;:8: Unused import - java.util.Arrays. [UnusedImports] ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2672: private RemoteIterator innerListStatus(Path f) throws FileNotFoundException,: Line is longer than 80 characters (found 93). [LineLength] ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2686: }:5: '}' at column 5 should be on the same line as the next part of a multi-block statement (one that directly contains multiple blocks: if/else-if/else, do/while or try/catch/finally). [RightCurly] ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java:45:import org.apache.hadoop.fs.s3a.impl.TestPartialDeleteFailures;:8: Unused import - org.apache.hadoop.fs.s3a.impl.TestPartialDeleteFailures. [UnusedImports] ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java:138: S3AFileStatus[] result = S3AUtils.iteratorToStatuses(resultItr, new HashSet<>());: Line is longer than 80 characters (found 85). [LineLength] ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java:205: S3AFileStatus[] result = S3AUtils.iteratorToStatuses(resultItr, new HashSet<>());: Line is longer than 80 characters (found 85). [LineLength] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17023) Tune listStatus() api of s3a.
[ https://issues.apache.org/jira/browse/HADOOP-17023?focusedWorklogId=480079=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480079 ] ASF GitHub Bot logged work on HADOOP-17023: --- Author: ASF GitHub Bot Created on: 08/Sep/20 12:04 Start Date: 08/Sep/20 12:04 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2257: URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688822205 checkstyle complaining with fixable issues ``` ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java:370: Listing.FileStatusListingIterator filesItr = createFileStatusListingIterator(: Line is longer than 80 characters (found 81). [LineLength] ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:34:import java.util.Arrays;:8: Unused import - java.util.Arrays. [UnusedImports] ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2672: private RemoteIterator innerListStatus(Path f) throws FileNotFoundException,: Line is longer than 80 characters (found 93). [LineLength] ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2686: }:5: '}' at column 5 should be on the same line as the next part of a multi-block statement (one that directly contains multiple blocks: if/else-if/else, do/while or try/catch/finally). [RightCurly] ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java:45:import org.apache.hadoop.fs.s3a.impl.TestPartialDeleteFailures;:8: Unused import - org.apache.hadoop.fs.s3a.impl.TestPartialDeleteFailures. [UnusedImports] ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java:138: S3AFileStatus[] result = S3AUtils.iteratorToStatuses(resultItr, new HashSet<>());: Line is longer than 80 characters (found 85). [LineLength] ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java:205: S3AFileStatus[] result = S3AUtils.iteratorToStatuses(resultItr, new HashSet<>());: Line is longer than 80 characters (found 85). [LineLength] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 480079) Time Spent: 1.5h (was: 1h 20m) > Tune listStatus() api of s3a. > - > > Key: HADOOP-17023 > URL: https://issues.apache.org/jira/browse/HADOOP-17023 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.1 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Similar optimisation which was done for listLocatedSttaus() > https://issues.apache.org/jira/browse/HADOOP-16465 can done for listStatus() > api as well. > This is going to reduce the number of remote calls in case of directory > listing. > > CC [~ste...@apache.org] [~shwethags] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17023) Tune listStatus() api of s3a.
[ https://issues.apache.org/jira/browse/HADOOP-17023?focusedWorklogId=480080=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480080 ] ASF GitHub Bot logged work on HADOOP-17023: --- Author: ASF GitHub Bot Created on: 08/Sep/20 12:04 Start Date: 08/Sep/20 12:04 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2257: URL: https://github.com/apache/hadoop/pull/2257#issuecomment-682588174 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 37m 46s | trunk passed | | +1 :green_heart: | compile | 1m 11s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 43s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 32s | trunk passed | | +1 :green_heart: | mvnsite | 0m 58s | trunk passed | | +1 :green_heart: | shadedclient | 20m 36s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 32s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 28s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 48s | the patch passed | | +1 :green_heart: | compile | 0m 48s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 48s | the patch passed | | +1 :green_heart: | compile | 0m 37s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 37s | the patch passed | | -0 :warning: | checkstyle | 0m 23s | hadoop-tools/hadoop-aws: The patch generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13) | | +1 :green_heart: | mvnsite | 0m 47s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 18m 48s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 18s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 5s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 1m 12s | The patch does not generate ASF License warnings. | | | | 96m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2257 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 1af3e1f76ff6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / bfd99f31fd6 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/testReport/ | | Max. process+thread count | 344 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/console | |
[jira] [Work logged] (HADOOP-17244) HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
[ https://issues.apache.org/jira/browse/HADOOP-17244?focusedWorklogId=480106=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480106 ] ASF GitHub Bot logged work on HADOOP-17244: --- Author: ASF GitHub Bot Created on: 08/Sep/20 13:04 Start Date: 08/Sep/20 13:04 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2280: URL: https://github.com/apache/hadoop/pull/2280#issuecomment-688851531 broke the local fs tests ``` [ERROR] Failures: [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameDirectoryAsEmptyDirectory:1192 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameDirectoryAsFile:1241 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameDirectoryAsNonEmptyDirectory:1216 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameDirectoryToItself:1112 Renamed directory to itself [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameDirectoryToNonExistentParent:1135 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameFileAsExistingDirectory:1093 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameFileAsExistingFile:1072 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameFileToDestinationWithParentFile:1018 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameFileToItself:1047 Renamed file to itself [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameFileToNonExistentDirectory:994 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameNonExistentPath:971 Should throw FileNotFoundException [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameDirectoryAsEmptyDirectory:1192 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameDirectoryAsFile:1241 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameDirectoryAsNonEmptyDirectory:1216 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameDirectoryToItself:1112 Renamed directory to itself [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameDirectoryToNonExistentParent:1135 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameFileAsExistingDirectory:1093 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameFileAsExistingFile:1072 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameFileToDestinationWithParentFile:1018 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameFileToItself:1047 Renamed file to itself [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameFileToNonExistentDirectory:994 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameNonExistentPath:971 Should throw FileNotFoundException [INFO] [ERROR] Tests run: 4439, Failures: 22, Errors: 0, Skipped: 165 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 480106) Time Spent: 40m (was: 0.5h) > HADOOP-17244. S3A directory delete tombstones dir markers prematurely. > -- > > Key: HADOOP-17244 > URL: https://issues.apache.org/jira/browse/HADOOP-17244 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Labels: pull-request-available >
[jira] [Work logged] (HADOOP-17113) Adding ReadAhead Counters in ABFS
[ https://issues.apache.org/jira/browse/HADOOP-17113?focusedWorklogId=480103=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480103 ] ASF GitHub Bot logged work on HADOOP-17113: --- Author: ASF GitHub Bot Created on: 08/Sep/20 13:03 Start Date: 08/Sep/20 13:03 Worklog Time Spent: 10m Work Description: steveloughran closed pull request #2282: URL: https://github.com/apache/hadoop/pull/2282 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 480103) Time Spent: 0.5h (was: 20m) > Adding ReadAhead Counters in ABFS > - > > Key: HADOOP-17113 > URL: https://issues.apache.org/jira/browse/HADOOP-17113 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Mehakmeet Singh >Assignee: Mehakmeet Singh >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Adding ReadAheads Counters in ABFS to track the behavior of the ReadAhead > feature in ABFS. This would include 2 counters: > |READ_AHEAD_BYTES_READ|number of bytes read by readAhead| > |READ_AHEAD_REMOTE_BYTES_READ|number of bytes not used after readAhead was > used| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17113) Adding ReadAhead Counters in ABFS
[ https://issues.apache.org/jira/browse/HADOOP-17113?focusedWorklogId=480105=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480105 ] ASF GitHub Bot logged work on HADOOP-17113: --- Author: ASF GitHub Bot Created on: 08/Sep/20 13:03 Start Date: 08/Sep/20 13:03 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2282: URL: https://github.com/apache/hadoop/pull/2282#issuecomment-688851051 merged to branch-3.3. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 480105) Time Spent: 40m (was: 0.5h) > Adding ReadAhead Counters in ABFS > - > > Key: HADOOP-17113 > URL: https://issues.apache.org/jira/browse/HADOOP-17113 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Mehakmeet Singh >Assignee: Mehakmeet Singh >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Adding ReadAheads Counters in ABFS to track the behavior of the ReadAhead > feature in ABFS. This would include 2 counters: > |READ_AHEAD_BYTES_READ|number of bytes read by readAhead| > |READ_AHEAD_REMOTE_BYTES_READ|number of bytes not used after readAhead was > used| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2280: HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
steveloughran commented on pull request #2280: URL: https://github.com/apache/hadoop/pull/2280#issuecomment-688851531 broke the local fs tests ``` [ERROR] Failures: [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameDirectoryAsEmptyDirectory:1192 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameDirectoryAsFile:1241 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameDirectoryAsNonEmptyDirectory:1216 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameDirectoryToItself:1112 Renamed directory to itself [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameDirectoryToNonExistentParent:1135 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameFileAsExistingDirectory:1093 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameFileAsExistingFile:1072 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameFileToDestinationWithParentFile:1018 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameFileToItself:1047 Renamed file to itself [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameFileToNonExistentDirectory:994 Expected exception was not thrown [ERROR] TestLocalFSFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameNonExistentPath:971 Should throw FileNotFoundException [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameDirectoryAsEmptyDirectory:1192 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameDirectoryAsFile:1241 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameDirectoryAsNonEmptyDirectory:1216 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameDirectoryToItself:1112 Renamed directory to itself [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameDirectoryToNonExistentParent:1135 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameFileAsExistingDirectory:1093 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameFileAsExistingFile:1072 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameFileToDestinationWithParentFile:1018 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameFileToItself:1047 Renamed file to itself [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameFileToNonExistentDirectory:994 Expected exception was not thrown [ERROR] TestFcMainOperationsLocalFs>FileContextMainOperationsBaseTest.testRenameNonExistentPath:971 Should throw FileNotFoundException [INFO] [ERROR] Tests run: 4439, Failures: 22, Errors: 0, Skipped: 165 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17229) Test failure as failed request body counted in byte received metric - ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
[ https://issues.apache.org/jira/browse/HADOOP-17229?focusedWorklogId=480119=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480119 ] ASF GitHub Bot logged work on HADOOP-17229: --- Author: ASF GitHub Bot Created on: 08/Sep/20 13:14 Start Date: 08/Sep/20 13:14 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2283: URL: https://github.com/apache/hadoop/pull/2283#issuecomment-688857463 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 30m 23s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-3.3 Compile Tests _ | | +1 :green_heart: | mvninstall | 34m 54s | branch-3.3 passed | | +1 :green_heart: | compile | 0m 36s | branch-3.3 passed | | +1 :green_heart: | checkstyle | 0m 25s | branch-3.3 passed | | +1 :green_heart: | mvnsite | 0m 42s | branch-3.3 passed | | +1 :green_heart: | shadedclient | 19m 14s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 28s | branch-3.3 passed | | +0 :ok: | spotbugs | 1m 6s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 3s | branch-3.3 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 36s | the patch passed | | +1 :green_heart: | compile | 0m 31s | the patch passed | | +1 :green_heart: | javac | 0m 31s | the patch passed | | +1 :green_heart: | checkstyle | 0m 19s | the patch passed | | +1 :green_heart: | mvnsite | 0m 32s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 18m 14s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | | +1 :green_heart: | findbugs | 1m 3s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 24s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. | | | | 113m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2283/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2283 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 123ec666bbb8 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-3.3 / 5710005 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2283/1/testReport/ | | Max. process+thread count | 422 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2283/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 480119) Time Spent: 2h 40m (was: 2.5h) > Test failure as failed request body counted in byte received metric - > ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics > --- > > Key: HADOOP-17229 > URL: https://issues.apache.org/jira/browse/HADOOP-17229 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, test >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Mehakmeet
[GitHub] [hadoop] hadoop-yetus commented on pull request #2283: HADOOP-17229. No updation of bytes received counter value after response failure occurs in ABFS
hadoop-yetus commented on pull request #2283: URL: https://github.com/apache/hadoop/pull/2283#issuecomment-688857463 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 30m 23s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-3.3 Compile Tests _ | | +1 :green_heart: | mvninstall | 34m 54s | branch-3.3 passed | | +1 :green_heart: | compile | 0m 36s | branch-3.3 passed | | +1 :green_heart: | checkstyle | 0m 25s | branch-3.3 passed | | +1 :green_heart: | mvnsite | 0m 42s | branch-3.3 passed | | +1 :green_heart: | shadedclient | 19m 14s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 28s | branch-3.3 passed | | +0 :ok: | spotbugs | 1m 6s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 3s | branch-3.3 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 36s | the patch passed | | +1 :green_heart: | compile | 0m 31s | the patch passed | | +1 :green_heart: | javac | 0m 31s | the patch passed | | +1 :green_heart: | checkstyle | 0m 19s | the patch passed | | +1 :green_heart: | mvnsite | 0m 32s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 18m 14s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | | +1 :green_heart: | findbugs | 1m 3s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 24s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. | | | | 113m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2283/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2283 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 123ec666bbb8 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-3.3 / 5710005 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2283/1/testReport/ | | Max. process+thread count | 422 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2283/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17229) Test failure as failed request body counted in byte received metric - ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
[
https://issues.apache.org/jira/browse/HADOOP-17229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-17229.
-
Fix Version/s: 3.4.0
Resolution: Fixed
> Test failure as failed request body counted in byte received metric -
> ITestAbfsNetworkStatistics#testAbfsHttpResponseStatistics
> ---
>
> Key: HADOOP-17229
> URL: https://issues.apache.org/jira/browse/HADOOP-17229
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, test
>Affects Versions: 3.3.0
>Reporter: Sneha Vijayarajan
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: abfsactive, pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Bytes received counter increments for every request response received.
> [https://github.com/apache/hadoop/blob/d23cc9d85d887f01d72180bdf1af87dfdee15c5a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java#L251]
> This increments even for failed requests.
> Observed during testing done for HADOOP-17215. A request failed with 409
> Conflict contains response body as below:
> {"error":\{"code":"PathAlreadyExists","message":"The specified path already
> exists.\nRequestId:c3b2c55c-b01f-0061-7b31-7b6ee300\nTime:2020-08-25T22:44:07.2356054Z"}}
> The error body of 168 size is incremented in bytes_received counter.
> This also breaks the testcase testAbfsHttpResponseStatistics.
> {code:java}
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR] Tests run: 2,
> Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 22.746 s <<< FAILURE! - in
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR]
> testAbfsHttpResponseStatistics(org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics)
> Time elapsed: 13.183 s <<< FAILURE!java.lang.AssertionError: Mismatch in
> bytes_received expected:<143> but was:<311> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.assertAbfsStatistics(AbstractAbfsIntegrationTest.java:445)
> at
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics.testAbfsHttpResponseStatistics(ITestAbfsNetworkStatistics.java:291)
> {code}
> [~mehakmeetSingh] - is the bytes_received counter increment for failed
> requests an expected behaviour ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16918) Dependency update for Hadoop 2.10
[ https://issues.apache.org/jira/browse/HADOOP-16918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192177#comment-17192177 ] Masatake Iwasaki commented on HADOOP-16918: --- Jetty was upgraded from 6.1.26 to 9.3.11.v20160721 HADOOP-10075. This is a big change which only fit for major upgrade (the fix version is 3.0.0-alpha2). > Dependency update for Hadoop 2.10 > - > > Key: HADOOP-16918 > URL: https://issues.apache.org/jira/browse/HADOOP-16918 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Priority: Major > Labels: release-blocker > Attachments: dependency-check-report.html > > > A number of dependencies can be updated. > nimbus-jose-jwt > jetty > netty > zookeeper > hbase-common > jackson-databind > and many more. They should be updated in the 2.10.1 release. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17113) Adding ReadAhead Counters in ABFS
[ https://issues.apache.org/jira/browse/HADOOP-17113?focusedWorklogId=480089=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480089 ] ASF GitHub Bot logged work on HADOOP-17113: --- Author: ASF GitHub Bot Created on: 08/Sep/20 12:18 Start Date: 08/Sep/20 12:18 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2282: URL: https://github.com/apache/hadoop/pull/2282#issuecomment-688828740 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 31m 4s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-3.3 Compile Tests _ | | +1 :green_heart: | mvninstall | 28m 33s | branch-3.3 passed | | +1 :green_heart: | compile | 0m 32s | branch-3.3 passed | | +1 :green_heart: | checkstyle | 0m 27s | branch-3.3 passed | | +1 :green_heart: | mvnsite | 0m 37s | branch-3.3 passed | | +1 :green_heart: | shadedclient | 15m 4s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 27s | branch-3.3 passed | | +0 :ok: | spotbugs | 0m 55s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 52s | branch-3.3 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 30s | the patch passed | | +1 :green_heart: | compile | 0m 24s | the patch passed | | +1 :green_heart: | javac | 0m 24s | the patch passed | | +1 :green_heart: | checkstyle | 0m 16s | the patch passed | | +1 :green_heart: | mvnsite | 0m 27s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 51s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | | +1 :green_heart: | findbugs | 0m 55s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 23s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 98m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2282/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2282 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 54bd4a666c1f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-3.3 / 5710005 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2282/1/testReport/ | | Max. process+thread count | 418 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2282/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 480089) Time Spent: 20m (was: 10m) > Adding ReadAhead Counters in ABFS > - > > Key: HADOOP-17113 > URL: https://issues.apache.org/jira/browse/HADOOP-17113 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Mehakmeet Singh >Assignee: Mehakmeet Singh >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Adding ReadAheads Counters
[jira] [Commented] (HADOOP-16942) S3A creating folder level delete markers
[
https://issues.apache.org/jira/browse/HADOOP-16942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192199#comment-17192199
]
Steve Loughran commented on HADOOP-16942:
-
one more point, you said: EMR version - 6.0.0
Amazon EMR's connector with S3 is their own closed source client. Please take
it up through your AWS support channel. thanks
> S3A creating folder level delete markers
>
>
> Key: HADOOP-16942
> URL: https://issues.apache.org/jira/browse/HADOOP-16942
> Project: Hadoop Common
> Issue Type: Task
> Components: fs/s3
>Affects Versions: 2.8.3, 3.2.1
>Reporter: vijayant soni
>Priority: Major
>
> Using S3A URL scheme while writing out data from Spark to S3 is creating many
> folder level delete markers.
> Writing the same with S3 URL scheme, does not create any delete markers at
> all.
>
> Spark - 2.4.4
> Hadoop - 3.2.1
> EMR version - 6.0.0
> Write Mode - Append
> {code:scala}
> [hadoop@ip-192-0-161-212 ~]$ spark-shell
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).
> 20/03/27 07:37:19 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive
> is set, falling back to uploading libraries under SPARK_HOME.
> Spark context Web UI available at http://ip-192-0-161-212.ec2.internal:4040
> Spark context available as 'sc' (master = yarn, app id =
> application_1585294390030_0003).
> Spark session available as 'spark'.
> Welcome to
> __
> / __/__ ___ _/ /__
> _\ \/ _ \/ _ `/ __/ '_/
>/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
> /_/
>
> Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_242)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala> val df = spark.sql("select 1 as a")
> df: org.apache.spark.sql.DataFrame = [a: int]
> scala>
> df.write.mode(org.apache.spark.sql.SaveMode.Append).save("s3://my-bucket/tmp/vijayant/test/s3/")
>
>
> scala>
> df.write.mode(org.apache.spark.sql.SaveMode.Append).save("s3a://my-bucket/tmp/vijayant/test/s3a/")
>
>
> scala>
> {code}
> Getting delete markers from `s3` write
> {code:bash}
> aws s3api list-object-versions --bucket my-bucket --prefix
> tmp/vijayant/test/s3/
> {
> "Versions": [
> {
> "LastModified": "2020-03-27T07:38:17.000Z",
> "VersionId": "V06OzeE7j221Tq7keSGj8bveCYyJFIcf",
> "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
> "StorageClass": "STANDARD",
> "Key": "tmp/vijayant/test/s3/_SUCCESS",
> "Owner": {
> "DisplayName": "sysops+stage",
> "ID":
> "08939105f417dc74b1fa237e211185ff2d9f528d54b1380501de07bd0657b5e1"
> },
> "IsLatest": true,
> "Size": 0
> },
> {
> "LastModified": "2020-03-27T07:38:16.000Z",
> "VersionId": "dLYtHDugLhFIdw2YHLFmoFOxXkm.21Wo",
> "ETag": "\"26e70a1e26c709e3e8498acd49cfaaa3-1\"",
> "StorageClass": "STANDARD",
> "Key":
> "tmp/vijayant/test/s3/part-0-9d9a8925-f119-415d-b547-b742396e2ca7-c000.snappy.parquet",
>
> "Owner": {
> "DisplayName": "sysops+stage",
> "ID":
> "08939105f417dc74b1fa237e211185ff2d9f528d54b1380501de07bd0657b5e1"
> },
> "IsLatest": true,
> "Size": 384
> }
> ]
> }
> {code}
> Getting delete markers from `s3a` write
> {code:bash}
> aws s3api list-object-versions --bucket my-bucket --prefix
> tmp/vijayant/test/s3a/
> {
> "DeleteMarkers": [
> {
> "Owner": {
> "DisplayName": "sysops+stage",
> "ID":
> "08939105f417dc74b1fa237e211185ff2d9f528d54b1380501de07bd0657b5e1"
> },
> "IsLatest": true,
> "VersionId": "NJWRZMcb_eYYwCJh_isX4H1Ox6W362Wb",
> "Key": "tmp/vijayant/test/s3a/",
> "LastModified": "2020-03-27T07:39:11.000Z"
> },
> {
> "Owner": {
> "DisplayName": "sysops+stage",
> "ID":
> "08939105f417dc74b1fa237e211185ff2d9f528d54b1380501de07bd0657b5e1"
> },
> "IsLatest": false,
> "VersionId": "F0h0mLcVVwkMtcHxd95Hj7BACL4Up_Q9",
> "Key": "tmp/vijayant/test/s3a/",
> "LastModified": "2020-03-27T07:39:10.000Z"
> },
> {
> "Owner": {
> "DisplayName": "sysops+stage",
> "ID":
>
[jira] [Commented] (HADOOP-16843) S3A DT marshalling to include nested error text in wrapped message
[ https://issues.apache.org/jira/browse/HADOOP-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192211#comment-17192211 ] Steve Loughran commented on HADOOP-16843: - Might be of interest for [~gabor.bota] > S3A DT marshalling to include nested error text in wrapped message > --- > > Key: HADOOP-16843 > URL: https://issues.apache.org/jira/browse/HADOOP-16843 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > > When you get an unmarshalling error with S3A DTs, the thrown exception wraps > the cause, but doesn't include it's text in the message, so on a long trace > the cause can get lost -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] iwasakims opened a new pull request #2285: HADOOP-17251. Upgrade netty-all to 4.1.50.Final on branch-2.10.
iwasakims opened a new pull request #2285: URL: https://github.com/apache/hadoop/pull/2285 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17251) Upgrade netty-all to 4.1.50.Final on branch-2.10
[ https://issues.apache.org/jira/browse/HADOOP-17251?focusedWorklogId=480153=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480153 ] ASF GitHub Bot logged work on HADOOP-17251: --- Author: ASF GitHub Bot Created on: 08/Sep/20 14:01 Start Date: 08/Sep/20 14:01 Worklog Time Spent: 10m Work Description: iwasakims opened a new pull request #2285: URL: https://github.com/apache/hadoop/pull/2285 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 480153) Remaining Estimate: 0h Time Spent: 10m > Upgrade netty-all to 4.1.50.Final on branch-2.10 > > > Key: HADOOP-17251 > URL: https://issues.apache.org/jira/browse/HADOOP-17251 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17241) s3a: bucket names which aren't parseable hostnames unsupported
[
https://issues.apache.org/jira/browse/HADOOP-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192236#comment-17192236
]
peter weissbrod commented on HADOOP-17241:
--
Further information: I've started to experience this issue in separate places.
Hive 3.0.0 metastore related issues.
Existing buckets (created march 2020) containing dots in the names. Creating
new/external table definitions using LOCATION 's3a://...; are successful (hive
seems to automatically scan the LOCATION for file stats upon creation. It makes
several low-level attempts but ultimately compiles a list of files and builds
statistics)
After creation all calls to SELECT from the creates tables fails with
{{InvalidInputException: Input path does not exist}}
In this scenario I attempt to load from the created table using pyspark v3
(running example below). My takeaway from this is that going forward s3a
support for buckets with dots will be flaky/unreliable at best.
For those of us that have long established s3 naming conventions using dots
this is a pervasive and serious development
{{Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.0.0
/_/
Using Scala version 2.12.10, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_121
Branch HEAD
Compiled by user ubuntu on 2020-06-06T11:32:25Z
Revision 3fdfce3120f307147244e5eaf46d61419a723d50
Url https://gitbox.apache.org/repos/asf/spark.git
# load a text file via sparkContext and ensure you can read it. This works!
>>> s3File = spark.sparkContext.textFile("s3a://mybucket.with.dots/_mydir/*")
>>> s3File.count()
1985
# attempt to (re) build a sql table from that same directory
>>> spark.sql('drop table if exists exp.baz')
DataFrame[]
>>> spark.sql("""
... create external table exp.baz (...) stored as textfile location
's3a://mybucket.with.dots/_mydir/'
""")
DataFrame[]
>>> spark.sql('select count(*) from exp.baz').show()
20/09/07 17:14:14 WARN DAGScheduler: Creating new stage failed due to exception
- job: 1
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:
s3a://mybucket.with.dots/_mydir
at
org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
at
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:205)
at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:276)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:272)
at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:276)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:272)
at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:276)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:272)
at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:276)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:272)
at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:276)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:272)
at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:276)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:272)
at org.apache.spark.rdd.RDD.getNumPartitions(RDD.scala:292)
at
org.apache.spark.scheduler.DAGScheduler.checkBarrierStageWithNumSlots(DAGScheduler.scala:435)
at
org.apache.spark.scheduler.DAGScheduler.createShuffleMapStage(DAGScheduler.scala:388)
at
org.apache.spark.scheduler.DAGScheduler.getOrCreateShuffleMapStage(DAGScheduler.scala:358)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$getOrCreateParentStages$1(DAGScheduler.scala:468)
at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at scala.collection.TraversableLike.map(TraversableLike.scala:238)
at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
at
[jira] [Work logged] (HADOOP-17242) S3A (async) ObjectListingIterator to block in hasNext() for results
[ https://issues.apache.org/jira/browse/HADOOP-17242?focusedWorklogId=480152=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480152 ] ASF GitHub Bot logged work on HADOOP-17242: --- Author: ASF GitHub Bot Created on: 08/Sep/20 14:01 Start Date: 08/Sep/20 14:01 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2273: URL: https://github.com/apache/hadoop/pull/2273#issuecomment-688895798 on this again. Need to think about what happens after hasNext raises an exception. IMO we should cache it and rethrow on all subsequent hasNext/next. Without that if you catch an IOE and call hasNext again it may get confused (maybe, I'd need to experiment) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 480152) Time Spent: 1h 10m (was: 1h) > S3A (async) ObjectListingIterator to block in hasNext() for results > --- > > Key: HADOOP-17242 > URL: https://issues.apache.org/jira/browse/HADOOP-17242 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > HADOOP-17074 made listing async in S3A, but the iterator's hasNext Call > doesn't wait for the result. If invoked on an empty path it *may* return when > it should be failing. > Note: surfaced in code review, not seen in the wild and all our tests were > happy -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2273: HADOOP-17242. S3A ObjectListingIterator to block in hasNext() for results
steveloughran commented on pull request #2273: URL: https://github.com/apache/hadoop/pull/2273#issuecomment-688895798 on this again. Need to think about what happens after hasNext raises an exception. IMO we should cache it and rethrow on all subsequent hasNext/next. Without that if you catch an IOE and call hasNext again it may get confused (maybe, I'd need to experiment) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17251) Upgrade netty-all to 4.1.50.Final on branch-2.10
[ https://issues.apache.org/jira/browse/HADOOP-17251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HADOOP-17251: -- Status: Patch Available (was: Open) > Upgrade netty-all to 4.1.50.Final on branch-2.10 > > > Key: HADOOP-17251 > URL: https://issues.apache.org/jira/browse/HADOOP-17251 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17251) Upgrade netty-all to 4.1.50.Final on branch-2.10
[ https://issues.apache.org/jira/browse/HADOOP-17251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HADOOP-17251: -- Description: netty-all seems to be easily updated to fix HADOOP-16918. > Upgrade netty-all to 4.1.50.Final on branch-2.10 > > > Key: HADOOP-17251 > URL: https://issues.apache.org/jira/browse/HADOOP-17251 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > netty-all seems to be easily updated to fix HADOOP-16918. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14910) Upgrade netty-all jar to latest 4.0.x.Final
[ https://issues.apache.org/jira/browse/HADOOP-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HADOOP-14910: -- Fix Version/s: 2.10.1 > Upgrade netty-all jar to latest 4.0.x.Final > --- > > Key: HADOOP-14910 > URL: https://issues.apache.org/jira/browse/HADOOP-14910 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Fix For: 3.1.0, 2.10.1 > > Attachments: HADOOP-14910-01.patch, HADOOP-14910-02.patch, > HADOOP-14910-03.patch > > > Upgrade netty-all jar to 4.0.37.Final version to fix latest vulnerabilities > reported. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14910) Upgrade netty-all jar to latest 4.0.x.Final
[ https://issues.apache.org/jira/browse/HADOOP-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1719#comment-1719 ] Masatake Iwasaki commented on HADOOP-14910: --- cherry-picked to branch-2.10. > Upgrade netty-all jar to latest 4.0.x.Final > --- > > Key: HADOOP-14910 > URL: https://issues.apache.org/jira/browse/HADOOP-14910 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Fix For: 3.1.0, 2.10.1 > > Attachments: HADOOP-14910-01.patch, HADOOP-14910-02.patch, > HADOOP-14910-03.patch > > > Upgrade netty-all jar to 4.0.37.Final version to fix latest vulnerabilities > reported. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17251) Upgrade netty-all to 4.1.50.Final on branch-2.10
Masatake Iwasaki created HADOOP-17251: - Summary: Upgrade netty-all to 4.1.50.Final on branch-2.10 Key: HADOOP-17251 URL: https://issues.apache.org/jira/browse/HADOOP-17251 Project: Hadoop Common Issue Type: Improvement Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
hadoop-yetus commented on pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688943301 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 32m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 31m 26s | trunk passed | | +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 26s | trunk passed | | +1 :green_heart: | mvnsite | 0m 37s | trunk passed | | +1 :green_heart: | shadedclient | 15m 53s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 26s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 1s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 0s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 30s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | compile | 0m 25s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 25s | the patch passed | | +1 :green_heart: | checkstyle | 0m 16s | the patch passed | | +1 :green_heart: | mvnsite | 0m 28s | the patch passed | | +1 :green_heart: | shellcheck | 0m 0s | There were no new shellcheck issues. | | +1 :green_heart: | shelldocs | 0m 14s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 15m 54s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 22s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 0s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 17s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. | | | | 108m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2278 | | Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml | | uname | Linux 476360760f4a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 0d855159f09 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/7/testReport/ | | Max. process+thread count | 309 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/7/console | | versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 | | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated.
[jira] [Work logged] (HADOOP-17251) Upgrade netty-all to 4.1.50.Final on branch-2.10
[ https://issues.apache.org/jira/browse/HADOOP-17251?focusedWorklogId=480174=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480174 ] ASF GitHub Bot logged work on HADOOP-17251: --- Author: ASF GitHub Bot Created on: 08/Sep/20 14:37 Start Date: 08/Sep/20 14:37 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2285: URL: https://github.com/apache/hadoop/pull/2285#issuecomment-688920889 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 11m 41s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-2.10 Compile Tests _ | | +1 :green_heart: | mvninstall | 17m 3s | branch-2.10 passed | | +1 :green_heart: | compile | 0m 19s | branch-2.10 passed | | +1 :green_heart: | mvnsite | 0m 24s | branch-2.10 passed | | +1 :green_heart: | javadoc | 0m 22s | branch-2.10 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 15s | the patch passed | | +1 :green_heart: | compile | 0m 13s | the patch passed | | +1 :green_heart: | javac | 0m 12s | the patch passed | | +1 :green_heart: | mvnsite | 0m 16s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 13s | hadoop-project in the patch passed. | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 34m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2285/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2285 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml | | uname | Linux a49ba550387b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-2.10 / 43c6c54 | | Default Java | Oracle Corporation-1.7.0_95-b00 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2285/1/testReport/ | | Max. process+thread count | 73 (vs. ulimit of 5500) | | modules | C: hadoop-project U: hadoop-project | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2285/1/console | | versions | git=2.7.4 maven=3.3.9 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 480174) Time Spent: 20m (was: 10m) > Upgrade netty-all to 4.1.50.Final on branch-2.10 > > > Key: HADOOP-17251 > URL: https://issues.apache.org/jira/browse/HADOOP-17251 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > netty-all seems to be easily updated to fix HADOOP-16918. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2285: HADOOP-17251. Upgrade netty-all to 4.1.50.Final on branch-2.10.
hadoop-yetus commented on pull request #2285: URL: https://github.com/apache/hadoop/pull/2285#issuecomment-688920889 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 11m 41s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-2.10 Compile Tests _ | | +1 :green_heart: | mvninstall | 17m 3s | branch-2.10 passed | | +1 :green_heart: | compile | 0m 19s | branch-2.10 passed | | +1 :green_heart: | mvnsite | 0m 24s | branch-2.10 passed | | +1 :green_heart: | javadoc | 0m 22s | branch-2.10 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 15s | the patch passed | | +1 :green_heart: | compile | 0m 13s | the patch passed | | +1 :green_heart: | javac | 0m 12s | the patch passed | | +1 :green_heart: | mvnsite | 0m 16s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 13s | hadoop-project in the patch passed. | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 34m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2285/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2285 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml | | uname | Linux a49ba550387b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-2.10 / 43c6c54 | | Default Java | Oracle Corporation-1.7.0_95-b00 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2285/1/testReport/ | | Max. process+thread count | 73 (vs. ulimit of 5500) | | modules | C: hadoop-project U: hadoop-project | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2285/1/console | | versions | git=2.7.4 maven=3.3.9 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on pull request #2285: HADOOP-17251. Upgrade netty-all to 4.1.50.Final on branch-2.10.
iwasakims commented on pull request #2285: URL: https://github.com/apache/hadoop/pull/2285#issuecomment-688930231 TestWebHDFS passed with the patch on my local. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17251) Upgrade netty-all to 4.1.50.Final on branch-2.10
[ https://issues.apache.org/jira/browse/HADOOP-17251?focusedWorklogId=480189=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480189 ] ASF GitHub Bot logged work on HADOOP-17251: --- Author: ASF GitHub Bot Created on: 08/Sep/20 14:51 Start Date: 08/Sep/20 14:51 Worklog Time Spent: 10m Work Description: iwasakims commented on pull request #2285: URL: https://github.com/apache/hadoop/pull/2285#issuecomment-688930231 TestWebHDFS passed with the patch on my local. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 480189) Time Spent: 0.5h (was: 20m) > Upgrade netty-all to 4.1.50.Final on branch-2.10 > > > Key: HADOOP-17251 > URL: https://issues.apache.org/jira/browse/HADOOP-17251 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > netty-all seems to be easily updated to fix HADOOP-16918. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #2284: HDFS-15561. Fix NullPointException when start dfsrouter
goiri commented on a change in pull request #2284:
URL: https://github.com/apache/hadoop/pull/2284#discussion_r485034706
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Router.java
##
@@ -553,11 +553,18 @@ public void verifyToken(DelegationTokenIdentifier
tokenId, byte[] password)
protected NamenodeHeartbeatService createLocalNamenodeHeartbeatService() {
// Detect NN running in this machine
String nsId = DFSUtil.getNamenodeNameServiceId(conf);
+
+if (nsId == null) {
+ LOG.error("Cannot find nameservice id for local {}", nsId);
Review comment:
Can we add a test for this situation?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16918) Dependency update for Hadoop 2.10
[ https://issues.apache.org/jira/browse/HADOOP-16918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192238#comment-17192238 ] Masatake Iwasaki commented on HADOOP-16918: --- I filed HADOOP-17251 to upgrade netty-all to 4.1.50.Final since it looks simple fix. > Dependency update for Hadoop 2.10 > - > > Key: HADOOP-16918 > URL: https://issues.apache.org/jira/browse/HADOOP-16918 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Priority: Major > Labels: release-blocker > Attachments: dependency-check-report.html > > > A number of dependencies can be updated. > nimbus-jose-jwt > jetty > netty > zookeeper > hbase-common > jackson-databind > and many more. They should be updated in the 2.10.1 release. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2284: HDFS-15561. Fix NullPointException when start dfsrouter
hadoop-yetus commented on pull request #2284: URL: https://github.com/apache/hadoop/pull/2284#issuecomment-688878940 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 34m 33s | trunk passed | | +1 :green_heart: | compile | 0m 41s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 37s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 27s | trunk passed | | +1 :green_heart: | mvnsite | 0m 40s | trunk passed | | +1 :green_heart: | shadedclient | 15m 18s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 38s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 54s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 13s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 12s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 30s | the patch passed | | +1 :green_heart: | compile | 0m 31s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 31s | the patch passed | | +1 :green_heart: | compile | 0m 29s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 29s | the patch passed | | +1 :green_heart: | checkstyle | 0m 17s | the patch passed | | +1 :green_heart: | mvnsite | 0m 31s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 1s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 33s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 50s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | findbugs | 1m 14s | hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 8m 28s | hadoop-hdfs-rbf in the patch passed. | | +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 86m 50s | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-rbf | | | Load of known null value in org.apache.hadoop.hdfs.server.federation.router.Router.createLocalNamenodeHeartbeatService() At Router.java:in org.apache.hadoop.hdfs.server.federation.router.Router.createLocalNamenodeHeartbeatService() At Router.java:[line 558] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2284/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2284 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux ccda73ed3592 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 0d855159f09 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2284/1/artifact/out/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2284/1/testReport/ | | Max. process+thread count | 2844 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
[jira] [Work logged] (HADOOP-17191) ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
[ https://issues.apache.org/jira/browse/HADOOP-17191?focusedWorklogId=480203=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480203 ] ASF GitHub Bot logged work on HADOOP-17191: --- Author: ASF GitHub Bot Created on: 08/Sep/20 15:10 Start Date: 08/Sep/20 15:10 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688943301 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 32m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 31m 26s | trunk passed | | +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 26s | trunk passed | | +1 :green_heart: | mvnsite | 0m 37s | trunk passed | | +1 :green_heart: | shadedclient | 15m 53s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 26s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 1s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 0s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 30s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | compile | 0m 25s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 25s | the patch passed | | +1 :green_heart: | checkstyle | 0m 16s | the patch passed | | +1 :green_heart: | mvnsite | 0m 28s | the patch passed | | +1 :green_heart: | shellcheck | 0m 0s | There were no new shellcheck issues. | | +1 :green_heart: | shelldocs | 0m 14s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 15m 54s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 22s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 0s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 17s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. | | | | 108m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2278 | | Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml | | uname | Linux 476360760f4a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 0d855159f09 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/7/testReport/ | | Max. process+thread count | 309
[jira] [Commented] (HADOOP-17181) ITestS3AContractUnbuffer failure -stream.read didn't return all data
[
https://issues.apache.org/jira/browse/HADOOP-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192297#comment-17192297
]
Steve Loughran commented on HADOOP-17181:
-
..and again. Maybe it's network load @ home or something
Also on Seek contract test happening on the same test run
{code}
[ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 35.016
s <<< FAILURE! - in
org.apache.hadoop.fs.contract.s3a.ITestS3AContractUnbuffer[ERROR]
testUnbufferBeforeRead(org.apache.hadoop.fs.contract.s3a.ITestS3AContractUnbuffer)
Time elapsed: 5.147 s <<< FAILURE! java.lang.AssertionError: failed to read
expected number of bytes from stream expected:<1024> but was:<515>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at
org.apache.hadoop.fs.contract.AbstractContractUnbufferTest.validateFileContents(AbstractContractUnbufferTest.java:139)
at
org.apache.hadoop.fs.contract.AbstractContractUnbufferTest.validateFullFileContents(AbstractContractUnbufferTest.java:132)
at
org.apache.hadoop.fs.contract.AbstractContractUnbufferTest.testUnbufferBeforeRead(AbstractContractUnbufferTest.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{code}
> ITestS3AContractUnbuffer failure -stream.read didn't return all data
>
>
> Key: HADOOP-17181
> URL: https://issues.apache.org/jira/browse/HADOOP-17181
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
>
> Seen 2x recently, failure in ITestS3AContractUnbuffer as not enough data came
> back in the read.
> The contract test assumes that stream.read() will return everything, but it
> could be some buffering problem. Proposed: switch to ReadFully to see if it
> is a quirk of the read/get or is something actually wrong with the production
> code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17251) Upgrade netty-all to 4.1.50.Final on branch-2.10
[ https://issues.apache.org/jira/browse/HADOOP-17251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17251: Labels: pull-request-available (was: ) > Upgrade netty-all to 4.1.50.Final on branch-2.10 > > > Key: HADOOP-17251 > URL: https://issues.apache.org/jira/browse/HADOOP-17251 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16918) Dependency update for Hadoop 2.10
[ https://issues.apache.org/jira/browse/HADOOP-16918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192241#comment-17192241 ] Masatake Iwasaki commented on HADOOP-16918: --- I would like to put zookeeper and hbase-common out of the scope of 2.10.1. [~weichiu] let me know if you think it (or other dependencies) should be blocker. > Dependency update for Hadoop 2.10 > - > > Key: HADOOP-16918 > URL: https://issues.apache.org/jira/browse/HADOOP-16918 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Priority: Major > Labels: release-blocker > Attachments: dependency-check-report.html > > > A number of dependencies can be updated. > nimbus-jose-jwt > jetty > netty > zookeeper > hbase-common > jackson-databind > and many more. They should be updated in the 2.10.1 release. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2281: HDFS-15516.Add info for create flags in NameNode audit logs.
hadoop-yetus commented on pull request #2281: URL: https://github.com/apache/hadoop/pull/2281#issuecomment-688967698 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 40s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 1s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 35m 6s | trunk passed | | +1 :green_heart: | compile | 1m 24s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 1m 13s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 54s | trunk passed | | +1 :green_heart: | mvnsite | 1m 24s | trunk passed | | +1 :green_heart: | shadedclient | 18m 36s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 51s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 17s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 3m 10s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 7s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 10s | the patch passed | | +1 :green_heart: | compile | 1m 13s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 1m 13s | the patch passed | | +1 :green_heart: | compile | 1m 4s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 1m 4s | the patch passed | | -0 :warning: | checkstyle | 0m 43s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 184 unchanged - 1 fixed = 187 total (was 185) | | +1 :green_heart: | mvnsite | 1m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 45s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 45s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 17s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 3m 12s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 117m 45s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | | | | 211m 49s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.server.namenode.TestDecommissioningStatus | | | hadoop.hdfs.server.namenode.TestNameNodeMXBean | | | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean | | | hadoop.hdfs.server.namenode.TestReencryptionWithKMS | | | hadoop.hdfs.server.datanode.TestBPOfferService | | | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.namenode.TestFsck | | | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2281/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2281 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 1e26f3632eda 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 0d855159f09 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2281/2/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit |
[jira] [Updated] (HADOOP-17250) ABFS: Allow random read sizes to be of buffer size
[ https://issues.apache.org/jira/browse/HADOOP-17250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17250: Affects Version/s: (was: 3.1.4) 3.3.0 > ABFS: Allow random read sizes to be of buffer size > -- > > Key: HADOOP-17250 > URL: https://issues.apache.org/jira/browse/HADOOP-17250 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: abfsactive > > ADLS Gen2/ABFS driver is optimized to read only the bytes that are requested > for when the read pattern is random. > It was observed in some spark jobs that though the reads are random, the next > read doesn't skip by a lot and can be served by the earlier read if read was > done in buffer size. As a result the job triggered a higher count of read > calls and resulted in higher job runtime. > When these jobs were run against Gen1 which always reads in buffer size , the > jobs fared well. > In this Jira we try to provide a control over config on random read to be of > requested size or buffer size. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17250) ABFS: Allow random read sizes to be of buffer size
[ https://issues.apache.org/jira/browse/HADOOP-17250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192286#comment-17192286 ] Steve Loughran commented on HADOOP-17250: - back-to-back readFully() are pretty common. Look @ S3AInputStream to see what is done there, which comes down to "have a minimum readahead range" > ABFS: Allow random read sizes to be of buffer size > -- > > Key: HADOOP-17250 > URL: https://issues.apache.org/jira/browse/HADOOP-17250 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.1.4 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: abfsactive > > ADLS Gen2/ABFS driver is optimized to read only the bytes that are requested > for when the read pattern is random. > It was observed in some spark jobs that though the reads are random, the next > read doesn't skip by a lot and can be served by the earlier read if read was > done in buffer size. As a result the job triggered a higher count of read > calls and resulted in higher job runtime. > When these jobs were run against Gen1 which always reads in buffer size , the > jobs fared well. > In this Jira we try to provide a control over config on random read to be of > requested size or buffer size. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2280: HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
hadoop-yetus commented on pull request #2280: URL: https://github.com/apache/hadoop/pull/2280#issuecomment-689012950 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 30m 39s | trunk passed | | +1 :green_heart: | compile | 24m 17s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 20m 30s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 2m 43s | trunk passed | | +1 :green_heart: | mvnsite | 2m 24s | trunk passed | | +1 :green_heart: | shadedclient | 19m 49s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 15s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 2m 11s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 21s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 34s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 25s | the patch passed | | +1 :green_heart: | compile | 19m 21s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 19m 21s | the patch passed | | +1 :green_heart: | compile | 17m 6s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 17m 6s | the patch passed | | +1 :green_heart: | checkstyle | 2m 44s | the patch passed | | +1 :green_heart: | mvnsite | 2m 23s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 9s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 13s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 2m 20s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 3m 47s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 42s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 1m 39s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 56s | The patch does not generate ASF License warnings. | | | | 189m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2280/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2280 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b43739be005f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 0d855159f09 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2280/2/testReport/ | | Max. process+thread count | 1554 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2280/2/console | | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 | | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
[jira] [Work logged] (HADOOP-17244) HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
[ https://issues.apache.org/jira/browse/HADOOP-17244?focusedWorklogId=480285=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480285 ] ASF GitHub Bot logged work on HADOOP-17244: --- Author: ASF GitHub Bot Created on: 08/Sep/20 17:03 Start Date: 08/Sep/20 17:03 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2280: URL: https://github.com/apache/hadoop/pull/2280#issuecomment-689012950 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 30m 39s | trunk passed | | +1 :green_heart: | compile | 24m 17s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 20m 30s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 2m 43s | trunk passed | | +1 :green_heart: | mvnsite | 2m 24s | trunk passed | | +1 :green_heart: | shadedclient | 19m 49s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 15s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 2m 11s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 21s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 34s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 25s | the patch passed | | +1 :green_heart: | compile | 19m 21s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 19m 21s | the patch passed | | +1 :green_heart: | compile | 17m 6s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 17m 6s | the patch passed | | +1 :green_heart: | checkstyle | 2m 44s | the patch passed | | +1 :green_heart: | mvnsite | 2m 23s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 9s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 13s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 2m 20s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 3m 47s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 42s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 1m 39s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 56s | The patch does not generate ASF License warnings. | | | | 189m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2280/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2280 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b43739be005f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 0d855159f09 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2280/2/testReport/ | | Max. process+thread count | 1554 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output |
[GitHub] [hadoop] sunchao commented on a change in pull request #2240: HADOOP-17165. Implement service-user feature in DecayRPCScheduler.
sunchao commented on a change in pull request #2240: URL: https://github.com/apache/hadoop/pull/2240#discussion_r485068150 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DecayRpcScheduler.java ## @@ -178,6 +188,7 @@ private static final double PRECISION = 0.0001; private MetricsProxy metricsProxy; private final CostProvider costProvider; + private Set serviceUsernames; Review comment: nit nit: usually it is `UserName` instead of `Username`, so better follow that pattern. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17165) Implement service-user feature in DecayRPCScheduler
[ https://issues.apache.org/jira/browse/HADOOP-17165?focusedWorklogId=480287=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480287 ] ASF GitHub Bot logged work on HADOOP-17165: --- Author: ASF GitHub Bot Created on: 08/Sep/20 17:04 Start Date: 08/Sep/20 17:04 Worklog Time Spent: 10m Work Description: sunchao commented on a change in pull request #2240: URL: https://github.com/apache/hadoop/pull/2240#discussion_r485068150 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/DecayRpcScheduler.java ## @@ -178,6 +188,7 @@ private static final double PRECISION = 0.0001; private MetricsProxy metricsProxy; private final CostProvider costProvider; + private Set serviceUsernames; Review comment: nit nit: usually it is `UserName` instead of `Username`, so better follow that pattern. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 480287) Time Spent: 1.5h (was: 1h 20m) > Implement service-user feature in DecayRPCScheduler > --- > > Key: HADOOP-17165 > URL: https://issues.apache.org/jira/browse/HADOOP-17165 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-17165.001.patch, HADOOP-17165.002.patch, > after.png, before.png > > Time Spent: 1.5h > Remaining Estimate: 0h > > In our cluster, we want to use FairCallQueue to limit heavy users, but not > want to restrict certain users who are submitting important requests. This > jira proposes to implement the service-user feature that the user is always > scheduled high-priority queue. > According to HADOOP-9640, the initial concept of FCQ has this feature, but > not implemented finally. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16918) Dependency update for Hadoop 2.10
[ https://issues.apache.org/jira/browse/HADOOP-16918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192377#comment-17192377 ] Wei-Chiu Chuang commented on HADOOP-16918: -- I should probably run this dependency checker again since that was generated 6 months ago. > Dependency update for Hadoop 2.10 > - > > Key: HADOOP-16918 > URL: https://issues.apache.org/jira/browse/HADOOP-16918 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Priority: Major > Labels: release-blocker > Attachments: dependency-check-report.html > > > A number of dependencies can be updated. > nimbus-jose-jwt > jetty > netty > zookeeper > hbase-common > jackson-databind > and many more. They should be updated in the 2.10.1 release. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
hadoop-yetus commented on pull request #2185: URL: https://github.com/apache/hadoop/pull/2185#issuecomment-689082045 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 56s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 29m 36s | trunk passed | | +1 :green_heart: | compile | 23m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 19m 47s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 3s | trunk passed | | +1 :green_heart: | mvnsite | 3m 1s | trunk passed | | +1 :green_heart: | shadedclient | 21m 55s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 31s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 27s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 3m 29s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 52s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 11s | the patch passed | | +1 :green_heart: | compile | 23m 34s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 23m 34s | the patch passed | | +1 :green_heart: | compile | 19m 49s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 19m 49s | the patch passed | | -0 :warning: | checkstyle | 2m 56s | root: The patch generated 5 new + 182 unchanged - 1 fixed = 187 total (was 183) | | +1 :green_heart: | mvnsite | 3m 0s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 16m 28s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 30s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 16s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 6m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 10m 33s | hadoop-common in the patch passed. | | -1 :x: | unit | 97m 51s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. | | | | 304m 7s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.server.mover.TestMover | | | hadoop.hdfs.server.namenode.TestProtectedDirectories | | | hadoop.hdfs.TestEncryptedTransfer | | | hadoop.hdfs.server.namenode.TestFileTruncate | | | hadoop.hdfs.server.namenode.TestReencryptionWithKMS | | | hadoop.hdfs.TestReconstructStripedFile | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.balancer.TestBalancer | | | hadoop.hdfs.TestDatanodeRegistration | | | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | | | hadoop.hdfs.server.namenode.TestCacheDirectives | | | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer | | | hadoop.hdfs.server.namenode.TestReencryption | | | hadoop.hdfs.TestFileChecksum | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/16/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2185 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux d118fba1e4d6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | |
[jira] [Commented] (HADOOP-17249) Upgrade jackson-databind to 2.10 on branch-2.10
[ https://issues.apache.org/jira/browse/HADOOP-17249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192311#comment-17192311 ] Jim Brennan commented on HADOOP-17249: -- [~iwasakims] there were some concerns about breaking downstream projects in [HADOOP-17094] https://issues.apache.org/jira/browse/HADOOP-17094?focusedCommentId=17145688=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17145688 > Upgrade jackson-databind to 2.10 on branch-2.10 > --- > > Key: HADOOP-17249 > URL: https://issues.apache.org/jira/browse/HADOOP-17249 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.10.0 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > This is filed to test backporting HADOOP-16905 to branch-2.10. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Gaurangi94 commented on a change in pull request #2028: MAPREDUCE-7279. Adds RM host name to history server web page
Gaurangi94 commented on a change in pull request #2028:
URL: https://github.com/apache/hadoop/pull/2028#discussion_r485081448
##
File path:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
##
@@ -1588,6 +1588,19 @@ public void stop() {
this.jobIndexInfo =
new JobIndexInfo(-1, -1, user, jobName, jobId, -1, -1, null,
queueName);
+
+ if (getConfig().getBoolean(
+ JHAdminConfig.MR_HISTORY_APPEND_RM_HOST_TO_HISTORY_FILE_NAME_ENABLED,
+
JHAdminConfig.DEFAULT_MR_HISTORY_APPEND_RM_HOST_TO_HISTORY_FILE_NAME_ENABLED)) {
+
+String hostName = getConfig().get(YarnConfiguration.RM_HOSTNAME, "");
Review comment:
I see that YARN federation feature can have multiple RMs.
I created a high availability cluster to check my changes, and noticed that
one of the RMs (job-cluster-m-0) was listed as the host.
Is it safe to assume that YARN federation will behave similarly? Please note
that I used Hadoop 3.2.1 to test this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16918) Dependency update for Hadoop 2.10
[ https://issues.apache.org/jira/browse/HADOOP-16918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192376#comment-17192376 ] Wei-Chiu Chuang commented on HADOOP-16918: -- We can probably update zookeeper from 3.4.9 to 3.4.14 to address all known vulnerabilities. Not sure if it's compatible. [~symat] any ideas? The hbase-common change requires a small update from 1.2.6 to 1.2.6.1. Seems like an easy one. > Dependency update for Hadoop 2.10 > - > > Key: HADOOP-16918 > URL: https://issues.apache.org/jira/browse/HADOOP-16918 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Priority: Major > Labels: release-blocker > Attachments: dependency-check-report.html > > > A number of dependencies can be updated. > nimbus-jose-jwt > jetty > netty > zookeeper > hbase-common > jackson-databind > and many more. They should be updated in the 2.10.1 release. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15891) Provide Regex Based Mount Point In Inode Tree
[ https://issues.apache.org/jira/browse/HADOOP-15891?focusedWorklogId=480379=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480379 ] ASF GitHub Bot logged work on HADOOP-15891: --- Author: ASF GitHub Bot Created on: 08/Sep/20 19:15 Start Date: 08/Sep/20 19:15 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2185: URL: https://github.com/apache/hadoop/pull/2185#issuecomment-689082045 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 56s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 29m 36s | trunk passed | | +1 :green_heart: | compile | 23m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 19m 47s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 3s | trunk passed | | +1 :green_heart: | mvnsite | 3m 1s | trunk passed | | +1 :green_heart: | shadedclient | 21m 55s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 31s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 27s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 3m 29s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 52s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 11s | the patch passed | | +1 :green_heart: | compile | 23m 34s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 23m 34s | the patch passed | | +1 :green_heart: | compile | 19m 49s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 19m 49s | the patch passed | | -0 :warning: | checkstyle | 2m 56s | root: The patch generated 5 new + 182 unchanged - 1 fixed = 187 total (was 183) | | +1 :green_heart: | mvnsite | 3m 0s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 16m 28s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 30s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 3m 16s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 6m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 10m 33s | hadoop-common in the patch passed. | | -1 :x: | unit | 97m 51s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. | | | | 304m 7s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.server.mover.TestMover | | | hadoop.hdfs.server.namenode.TestProtectedDirectories | | | hadoop.hdfs.TestEncryptedTransfer | | | hadoop.hdfs.server.namenode.TestFileTruncate | | | hadoop.hdfs.server.namenode.TestReencryptionWithKMS | | | hadoop.hdfs.TestReconstructStripedFile | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.balancer.TestBalancer | | | hadoop.hdfs.TestDatanodeRegistration | | | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | | | hadoop.hdfs.server.namenode.TestCacheDirectives | | | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer | | | hadoop.hdfs.server.namenode.TestReencryption | | | hadoop.hdfs.TestFileChecksum | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base:
[jira] [Work logged] (HADOOP-15702) ABFS: Increase timeout of ITestAbfsReadWriteAndSeek
[ https://issues.apache.org/jira/browse/HADOOP-15702?focusedWorklogId=480264=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-480264 ] ASF GitHub Bot logged work on HADOOP-15702: --- Author: ASF GitHub Bot Created on: 08/Sep/20 16:28 Start Date: 08/Sep/20 16:28 Worklog Time Spent: 10m Work Description: bilaharith closed pull request #2152: URL: https://github.com/apache/hadoop/pull/2152 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 480264) Remaining Estimate: 0h Time Spent: 10m > ABFS: Increase timeout of ITestAbfsReadWriteAndSeek > --- > > Key: HADOOP-15702 > URL: https://issues.apache.org/jira/browse/HADOOP-15702 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, test >Affects Versions: HADOOP-15407, 3.4.0 >Reporter: Sean Mackrory >Assignee: Bilahari T H >Priority: Major > Labels: abfsinactive > Time Spent: 10m > Remaining Estimate: 0h > > ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek > fails for me all the time. Let's increase the timout limit. > It also seems to get executed twice... -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15702) ABFS: Increase timeout of ITestAbfsReadWriteAndSeek
[ https://issues.apache.org/jira/browse/HADOOP-15702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-15702: Labels: abfsinactive pull-request-available (was: abfsinactive) > ABFS: Increase timeout of ITestAbfsReadWriteAndSeek > --- > > Key: HADOOP-15702 > URL: https://issues.apache.org/jira/browse/HADOOP-15702 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, test >Affects Versions: HADOOP-15407, 3.4.0 >Reporter: Sean Mackrory >Assignee: Bilahari T H >Priority: Major > Labels: abfsinactive, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek > fails for me all the time. Let's increase the timout limit. > It also seems to get executed twice... -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
