[GitHub] [hadoop] hadoop-yetus commented on issue #1284: HDDS-1952. Disable TestMiniChaosOzoneCluster in integration.sh
hadoop-yetus commented on issue #1284: HDDS-1952. Disable TestMiniChaosOzoneCluster in integration.sh URL: https://github.com/apache/hadoop/pull/1284#issuecomment-520720481 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 41 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | 0 | shelldocs | 0 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 640 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 783 | branch has no errors when building and testing our client artifacts. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 574 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 0 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 678 | patch has no errors when building and testing our client artifacts. | ||| _ Other Tests _ | | +1 | unit | 103 | hadoop-hdds in the patch passed. | | +1 | unit | 286 | hadoop-ozone in the patch passed. | | +1 | asflicense | 50 | The patch does not generate ASF License warnings. | | | | 3341 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1284/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1284 | | Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs | | uname | Linux e67b550e84a9 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 454420e | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1284/2/testReport/ | | Max. process+thread count | 447 (vs. ulimit of 5500) | | modules | C: hadoop-ozone U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1284/2/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16403) Start a new statistical rpc queue and make the Reader's pendingConnection queue runtime-replaceable
[
https://issues.apache.org/jira/browse/HADOOP-16403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905897#comment-16905897
]
Hadoop QA commented on HADOOP-16403:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 18m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
44s{color} | {color:green} root: The patch generated 0 new + 544 unchanged - 11
fixed = 544 total (was 555) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
58s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}106m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}234m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReconstructStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HADOOP-16403 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977434/HADOOP-16403.008.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle cc |
| uname | Linux 9361c8bc3074 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
|
[jira] [Commented] (HADOOP-16377) Moving logging APIs over to slf4j
[
https://issues.apache.org/jira/browse/HADOOP-16377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905905#comment-16905905
]
Prabhu Joseph commented on HADOOP-16377:
[~jojochuang] [~ste...@apache.org] Can you review this Jira when you get time.
Thanks.
> Moving logging APIs over to slf4j
> -
>
> Key: HADOOP-16377
> URL: https://issues.apache.org/jira/browse/HADOOP-16377
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: HADOOP-16357-002.patch, HADOOP-16377-001.patch,
> HADOOP-16377-003.patch, HADOOP-16377-004.patch, HADOOP-16377-005.patch,
> HADOOP-16377-006.patch, HADOOP-16377-007.patch, HADOOP-16377-008.patch,
> HADOOP-16377-009.patch
>
>
> As of today, there are still 50 references to log4j1
> {code}
> $ grep -r "import org.apache.commons.logging.Log;" . |wc - l
> 50
> {code}
> To achieve the goal of HADOOP-12956/HADOOP-16206, we should invest time to
> move them to slf4j
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905932#comment-16905932 ] Jinglun commented on HADOOP-15565: -- Hi [~jojochuang] [~xkrogen] [~gabor.bota] [~hexiaoqiao], patch-005 is ready now. Could you help review it, thanks! > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, > HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, HADOOP-15565.0005.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16403) Start a new statistical rpc queue and make the Reader's pendingConnection queue runtime-replaceable
[
https://issues.apache.org/jira/browse/HADOOP-16403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905934#comment-16905934
]
Jinglun commented on HADOOP-16403:
--
Hi [~jojochuang] [~xkrogen], patch-008 is ready now. Do you have time to review
it? Thanks!:)
> Start a new statistical rpc queue and make the Reader's pendingConnection
> queue runtime-replaceable
> ---
>
> Key: HADOOP-16403
> URL: https://issues.apache.org/jira/browse/HADOOP-16403
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HADOOP-16403-How_MetricLinkedBlockingQueue_Works.pdf,
> HADOOP-16403.001.patch, HADOOP-16403.002.patch, HADOOP-16403.003.patch,
> HADOOP-16403.004.patch, HADOOP-16403.005.patch, HADOOP-16403.006.patch,
> HADOOP-16403.007.patch, HADOOP-16403.008.patch,
> MetricLinkedBlockingQueueTest.pdf
>
>
> I have an HA cluster with 2 NameNodes. The NameNode's meta is quite big so
> after the active dead, it takes the standby more than 40s to become active.
> Many requests(tcp connect request and rpc request) from Datanodes, clients
> and zkfc timed out and start retrying. The suddenly request flood lasts for
> the next 2 minutes and finally all requests are either handled or run out of
> retry times.
> Adjusting the rpc related settings might power the NameNode and solve this
> problem and the key point is finding the bottle neck. The rpc server can be
> described as below:
> {noformat}
> Listener -> Readers' queues -> Readers -> callQueue -> Handlers{noformat}
> By sampling some failed clients, I find many of them got
> ConnectTimeoutException. It's caused by a 20s un-responded tcp connect
> request. I think may be the reader queue is full and block the listener from
> handling new connections. Both slow handlers and slow readers can block the
> whole processing progress, and I need to know who it is. I think *a queue
> that computes the qps, write log when the queue is full and could be replaced
> easily* will help.
> I find the nice work HADOOP-10302 implementing a runtime-swapped queue.
> Using it at Reader's queue makes the reader queue runtime-swapped
> automatically. The qps computing job could be done by implementing a subclass
> of LinkedBlockQueue that does the computing job while put/take/... happens.
> The qps data will show on jmx.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai opened a new pull request #1287: HDDS-1956. Aged IO Thread exits on first read
adoroszlai opened a new pull request #1287: HDDS-1956. Aged IO Thread exits on first read URL: https://github.com/apache/hadoop/pull/1287 ## What changes were proposed in this pull request? 1. Aged IO thread exits on first read due to `ArrayIndexOutOfBoundsException`. This is caused by using `index` as the key name, while `readKey` expects a string separated by `_`. 2. After fixing the above, it randomly exits due to `OMException: Key not found`. This is an off-by-one error. The first key written has `index=1`, but the test randomly attempts to read `index=0`. 3. Add trace level message for read/write/delete operations. 4. Include thread name in log output pattern for `integration-test`. 5. Fix some log messages in `MiniOzoneChaosCluster` https://issues.apache.org/jira/browse/HDDS-1956 ## How was this patch tested? Manually ran `TestMiniChaosOzoneCluster`: ``` mvn -Phdds -pl :hadoop-ozone-integration-test -Dtest=TestMiniChaosOzoneCluster test ``` Stopped it after a few minutes (see [HDDS-1952](https://issues.apache.org/jira/browse/HDDS-1952)). Verified that aged IO thread was writing and reading keys (until it started failing due to chaos): ``` 2019-08-13 10:19:18,307 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(138)) - LOADGEN: Writing key pool-245-thread-1_0 2019-08-13 10:19:18,433 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(143)) - LOADGEN: Written key pool-245-thread-1_0 2019-08-13 10:19:22,473 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - LOADGEN: Reading key pool-245-thread-1_0 2019-08-13 10:19:22,992 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - LOADGEN: Read key pool-245-thread-1_0 ... 2019-08-13 10:19:23,837 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - LOADGEN: Reading key pool-245-thread-1_0 2019-08-13 10:19:23,866 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - LOADGEN: Read key pool-245-thread-1_0 2019-08-13 10:19:23,866 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(138)) - LOADGEN: Writing key pool-245-thread-1_1 2019-08-13 10:19:23,870 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(143)) - LOADGEN: Written key pool-245-thread-1_1 2019-08-13 10:19:23,891 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - LOADGEN: Reading key pool-245-thread-1_1 2019-08-13 10:19:23,933 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - LOADGEN: Read key pool-245-thread-1_1 2019-08-13 10:19:23,933 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - LOADGEN: Reading key pool-245-thread-1_0 2019-08-13 10:19:23,960 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - LOADGEN: Read key pool-245-thread-1_0 2019-08-13 10:19:23,960 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - LOADGEN: Reading key pool-245-thread-1_0 2019-08-13 10:19:23,974 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - LOADGEN: Read key pool-245-thread-1_0 2019-08-13 10:19:23,974 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - LOADGEN: Reading key pool-245-thread-1_0 2019-08-13 10:19:24,017 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - LOADGEN: Read key pool-245-thread-1_0 2019-08-13 10:19:24,017 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - LOADGEN: Reading key pool-245-thread-1_1 2019-08-13 10:19:24,044 [pool-245-thread-1] TRACE ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - LOADGEN: Read key pool-245-thread-1_1 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1287: HDDS-1956. Aged IO Thread exits on first read
adoroszlai commented on issue #1287: HDDS-1956. Aged IO Thread exits on first read URL: https://github.com/apache/hadoop/pull/1287#issuecomment-520743106 @mukul1987 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905993#comment-16905993
]
Adam Antal commented on HADOOP-16503:
-
Indeed, sorry.
I remember uploading it and then closing the tab - the upload might have
failed, my bad. Uploaded: [^HADOOP-16503.001.patch].
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16503.001.patch
>
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated HADOOP-16503:
Attachment: HADOOP-16503.001.patch
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16503.001.patch
>
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905995#comment-16905995
]
Adam Antal commented on HADOOP-16503:
-
For the verification of the patch one may use the following command:
{noformat}
mvn clean package -Dit.test=None
-Dtest=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
{noformat}
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16503.001.patch
>
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1273: HDDS-1951. Wrong symbolic release name on 0.4.1 branch
adoroszlai commented on issue #1273: HDDS-1951. Wrong symbolic release name on 0.4.1 branch URL: https://github.com/apache/hadoop/pull/1273#issuecomment-520761720 @nandakumar131 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on a change in pull request #1281: HDDS-1955. TestBlockOutputStreamWithFailures#test2DatanodesFailure failing because of assertion error. Contributed by Mukul K
bshashikant commented on a change in pull request #1281: HDDS-1955.
TestBlockOutputStreamWithFailures#test2DatanodesFailure failing because of
assertion error. Contributed by Mukul Kumar Singh.
URL: https://github.com/apache/hadoop/pull/1281#discussion_r313302193
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailures.java
##
@@ -501,14 +501,14 @@ public void test2DatanodesFailure() throws Exception {
// and one flush for partial chunk
key.flush();
+Throwable ioException = HddsClientUtils.checkForException(
+blockOutputStream.getIoException());
// Since, 2 datanodes went down, if the pipeline gets destroyed quickly,
// it will hit GroupMismatchException else, it will fail with
// RaftRetryFailureException
Review comment:
Please update the comment stating it can also fail with container not open
exception in case the containers got closed quickly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on issue #1281: HDDS-1955. TestBlockOutputStreamWithFailures#test2DatanodesFailure failing because of assertion error. Contributed by Mukul Kumar Singh.
bshashikant commented on issue #1281: HDDS-1955. TestBlockOutputStreamWithFailures#test2DatanodesFailure failing because of assertion error. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/1281#issuecomment-520763229 The patch overall looks good to me. I have just added a minor comment to update the comments in the test. I am +1 after that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1284: HDDS-1952. Disable TestMiniChaosOzoneCluster in integration.sh
adoroszlai commented on issue #1284: HDDS-1952. Disable TestMiniChaosOzoneCluster in integration.sh URL: https://github.com/apache/hadoop/pull/1284#issuecomment-520769904 Thanks @mukul1987 for reviewing it. Unfortunately `excludes` only works in `pom.xml`, not in the command-line. The updated patch successfully excluded `TestMiniChaosOzoneCluster` in the [CI run](https://ci.anzix.net/job/ozone/17657/consoleText). As a side-effect, it also ran all tests in `hadoop-ozone-filesystem` (see #1235 for a more complete fix for [HDDS-1916](https://issues.apache.org/jira/browse/HDDS-1916)). Acceptance test failure is unrelated ([HDDS-1936](https://issues.apache.org/jira/browse/HDDS-1936)). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16264) [JDK11] Track failing Hadoop unit tests
[ https://issues.apache.org/jira/browse/HADOOP-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906029#comment-16906029 ] Adam Antal commented on HADOOP-16264: - Created a few Yarn-related test fixing jiras: HADOOP-16503, YARN-9741, YARN-9742, YARN-9743 as a starting point. I want to get through and fix them one by one. Will also post the results of the UTs on openjdk-11.0.4 soon focusing on the Yarn part. > [JDK11] Track failing Hadoop unit tests > --- > > Key: HADOOP-16264 > URL: https://issues.apache.org/jira/browse/HADOOP-16264 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.1.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: test-run1.tgz, test-run2.log, test-run2.log.gz, > test-run3-openjdk11.0.3u7-ubuntu19.04.log > > > Although there are still a lot of work before we could compile Hadoop with > JDK 11 (HADOOP-15338), it is possible to compile Hadoop with JDK 8 and run > (e.g. HDFS NN/DN,YARN NM/RM) on JDK 11 at this moment. > But after compiling branch-3.1.2 with JDK 8, I ran unit tests with JDK 11 and > there are a LOT of unit test failures (44 out of 96 maven projects contain at > least one unit test failures according to maven reactor summary). This may > well indicate some functionalities are actually broken on JDK 11. Some of > them already have a jira number. Some of them might have been fixed in 3.2.0. > Some of them might share the same root cause. > By definition, this jira should be part of HADOOP-15338. But the goal of this > one is just to keep track of unit test failures and (hopefully) resolve all > of them soon. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 closed pull request #1273: HDDS-1951. Wrong symbolic release name on 0.4.1 branch
nandakumar131 closed pull request #1273: HDDS-1951. Wrong symbolic release name on 0.4.1 branch URL: https://github.com/apache/hadoop/pull/1273 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on issue #1282: HDDS-1908. TestMultiBlockWritesWithDnFailures is failing
bshashikant commented on issue #1282: HDDS-1908. TestMultiBlockWritesWithDnFailures is failing URL: https://github.com/apache/hadoop/pull/1282#issuecomment-520773739 Thanks @adoroszlai for working on this. The change looks good to me. +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant merged pull request #1282: HDDS-1908. TestMultiBlockWritesWithDnFailures is failing
bshashikant merged pull request #1282: HDDS-1908. TestMultiBlockWritesWithDnFailures is failing URL: https://github.com/apache/hadoop/pull/1282 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1234: HDDS-1917. TestOzoneRpcClientAbstract is failing.
hadoop-yetus commented on issue #1234: HDDS-1917. TestOzoneRpcClientAbstract is failing. URL: https://github.com/apache/hadoop/pull/1234#issuecomment-520777483 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 73 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 619 | trunk passed | | +1 | compile | 390 | trunk passed | | +1 | checkstyle | 77 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 958 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 170 | trunk passed | | 0 | spotbugs | 447 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 646 | trunk passed | | -0 | patch | 490 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 587 | the patch passed | | +1 | compile | 377 | the patch passed | | +1 | javac | 377 | the patch passed | | +1 | checkstyle | 76 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 777 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 171 | the patch passed | | +1 | findbugs | 709 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 343 | hadoop-hdds in the patch passed. | | -1 | unit | 2565 | hadoop-ozone in the patch failed. | | +1 | asflicense | 42 | The patch does not generate ASF License warnings. | | | | 8747 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.om.TestSecureOzoneManager | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1234 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 58ea09cb5782 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 454420e | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/6/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/6/testReport/ | | Max. process+thread count | 5123 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/integration-test U: hadoop-ozone/integration-test | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/6/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15616) Incorporate Tencent Cloud COS File System Implementation
[ https://issues.apache.org/jira/browse/HADOOP-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906051#comment-16906051 ] Junping Du commented on HADOOP-15616: - [~Sammi], could you help to drive this review work? I don't have bandwidth to do it recently. > Incorporate Tencent Cloud COS File System Implementation > > > Key: HADOOP-15616 > URL: https://issues.apache.org/jira/browse/HADOOP-15616 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/cos >Reporter: Junping Du >Assignee: YangY >Priority: Major > Attachments: HADOOP-15616.001.patch, HADOOP-15616.002.patch, > HADOOP-15616.003.patch, HADOOP-15616.004.patch, HADOOP-15616.005.patch, > HADOOP-15616.006.patch, HADOOP-15616.007.patch, HADOOP-15616.008.patch, > HADOOP-15616.009.patch, HADOOP-15616.010.patch, > Tencent-COS-Integrated-v2.pdf, Tencent-COS-Integrated.pdf > > > Tencent cloud is top 2 cloud vendors in China market and the object store COS > （[https://intl.cloud.tencent.com/product/cos]） is widely used among China’s > cloud users but now it is hard for hadoop user to access data laid on COS > storage as no native support for COS in Hadoop. > This work aims to integrate Tencent cloud COS with Hadoop/Spark/Hive, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from COS without any code change. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1282: HDDS-1908. TestMultiBlockWritesWithDnFailures is failing
adoroszlai commented on issue #1282: HDDS-1908. TestMultiBlockWritesWithDnFailures is failing URL: https://github.com/apache/hadoop/pull/1282#issuecomment-520781050 Thank you @bshashikant for reviewing and committing it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1287: HDDS-1956. Aged IO Thread exits on first read
hadoop-yetus commented on issue #1287: HDDS-1956. Aged IO Thread exits on first read URL: https://github.com/apache/hadoop/pull/1287#issuecomment-520783651 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 42 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 614 | trunk passed | | +1 | compile | 368 | trunk passed | | +1 | checkstyle | 65 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 842 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 167 | trunk passed | | 0 | spotbugs | 427 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 626 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 560 | the patch passed | | +1 | compile | 376 | the patch passed | | +1 | javac | 376 | the patch passed | | +1 | checkstyle | 80 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 684 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 171 | the patch passed | | +1 | findbugs | 651 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 290 | hadoop-hdds in the patch passed. | | -1 | unit | 1660 | hadoop-ozone in the patch failed. | | +1 | asflicense | 52 | The patch does not generate ASF License warnings. | | | | 7432 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1287 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux abd4695b6acf 4.4.0-157-generic #185-Ubuntu SMP Tue Jul 23 09:17:01 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 454420e | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/1/testReport/ | | Max. process+thread count | 5125 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/integration-test U: hadoop-ozone/integration-test | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg edited a comment on issue #1208: HADOOP-16423. S3Guard fsck: Check metadata consistency between S3 and metadatastore (log)
bgaborg edited a comment on issue #1208: HADOOP-16423. S3Guard fsck: Check metadata consistency between S3 and metadatastore (log) URL: https://github.com/apache/hadoop/pull/1208#issuecomment-520787365 - [x] - Fixed most of the checkstyle issues, however I left the long line length as it is because I think it's more readable this way. - [ ] - Fixed findbugs issues. - [ ] - Added more, and fixed comments. - [ ] - Created several subtasks in jira to add more features to this improvement. - [ ] Please review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg edited a comment on issue #1208: HADOOP-16423. S3Guard fsck: Check metadata consistency between S3 and metadatastore (log)
bgaborg edited a comment on issue #1208: HADOOP-16423. S3Guard fsck: Check metadata consistency between S3 and metadatastore (log) URL: https://github.com/apache/hadoop/pull/1208#issuecomment-520787365 - [x] - Fixed most of the checkstyle issues, however I left the long line length as it is because I think it's more readable this way. - [x] - Fixed findbugs issues. - [x] - Added more, and fixed comments. - [x] - Created several subtasks in jira to add more features to this improvement. - [ ] Please review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg edited a comment on issue #1208: HADOOP-16423. S3Guard fsck: Check metadata consistency between S3 and metadatastore (log)
bgaborg edited a comment on issue #1208: HADOOP-16423. S3Guard fsck: Check metadata consistency between S3 and metadatastore (log) URL: https://github.com/apache/hadoop/pull/1208#issuecomment-520787365 - [x] - Fixed most of the checkstyle issues, however I left the long line length as it is because I think it's more readable this way. - [x] - Fixed findbugs issues. - [x] - Added more, and fixed comments. - [ ] - Created several subtasks in jira to add more features to this improvement. - [ ] Please review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg edited a comment on issue #1208: HADOOP-16423. S3Guard fsck: Check metadata consistency between S3 and metadatastore (log)
bgaborg edited a comment on issue #1208: HADOOP-16423. S3Guard fsck: Check metadata consistency between S3 and metadatastore (log) URL: https://github.com/apache/hadoop/pull/1208#issuecomment-520787365 - [x] - Fixed most of the checkstyle issues, however I left the long line length as it is because I think it's more readable this way. - [x] - Fixed findbugs issues. - [ ] - Added more, and fixed comments. - [ ] - Created several subtasks in jira to add more features to this improvement. - [ ] Please review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on issue #1208: HADOOP-16423. S3Guard fsck: Check metadata consistency between S3 and metadatastore (log)
bgaborg commented on issue #1208: HADOOP-16423. S3Guard fsck: Check metadata consistency between S3 and metadatastore (log) URL: https://github.com/apache/hadoop/pull/1208#issuecomment-520787365 - [ ] - Fixed most of the checkstyle issues, however I left the long line length as it is because I think it's more readable this way. - [ ] - Fixed findbugs issues. - [ ] - Added more, and fixed comments. - [ ] - Created several subtasks in jira to add more features to this improvement. - [ ] Please review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 commented on issue #1234: HDDS-1917. TestOzoneRpcClientAbstract is failing.
nandakumar131 commented on issue #1234: HDDS-1917. TestOzoneRpcClientAbstract is failing. URL: https://github.com/apache/hadoop/pull/1234#issuecomment-520797332 @xiaoyuyao The problem is that we are not considering `OzoneAclType` while checking if a OzoneKey already has the ACL or not. In build machines we have user and group with same name (jenkins), so the OzoneKey have ACLs with same name but different type (user & group). Since we weren't checking `OzoneAclType` we were updating and removing wrong ACL for the key. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16492) Support HuaweiCloud Object Storage - as a file system in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjun updated HADOOP-16492: -- Attachment: huaweicloud-obs-integrate.pdf > Support HuaweiCloud Object Storage - as a file system in Hadoop > --- > > Key: HADOOP-16492 > URL: https://issues.apache.org/jira/browse/HADOOP-16492 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 3.3.0 >Reporter: zhongjun >Priority: Major > Attachments: huaweicloud-obs-integrate.pdf > > > Added support for HuaweiCloud > OBS([https://www.huaweicloud.com/en-us/product/obs.html]) to Hadoop, just > like what we do before for S3, ADL, OSS, etc. With simple configuration, > Hadoop applications can read/write data from OBS without any code change. > > h2. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906132#comment-16906132
]
Hadoop QA commented on HADOOP-16503:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 387 unchanged - 0 fixed = 388 total (was 387)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 59s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HADOOP-16503 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977460/HADOOP-16503.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7a19a9419ac8 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 454420e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16474/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreComm
[GitHub] [hadoop] lokeshj1703 commented on a change in pull request #1279: HDDS-1942. Support copy during S3 multipart upload part creation
lokeshj1703 commented on a change in pull request #1279: HDDS-1942. Support
copy during S3 multipart upload part creation
URL: https://github.com/apache/hadoop/pull/1279#discussion_r313367705
##
File path:
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##
@@ -736,4 +761,25 @@ private CopyObjectResponse copyObject(String copyHeader,
}
}
}
+
+ /**
+ * Parse the key and bucket name from copy header.
+ */
+ private Pair parseSourceHeader(String copyHeader)
+ throws OS3Exception {
+String header = copyHeader;
+if (header.startsWith("/")) {
+ header = copyHeader.substring(1);
+}
+int pos = header.indexOf("/");
+if (pos == -1) {
+ OS3Exception ex = S3ErrorTable.newError(S3ErrorTable
+ .INVALID_ARGUMENT, header);
+ ex.setErrorMessage("Copy Source must mention the source bucket and " +
+ "key: sourcebucket/sourcekey");
+ throw ex;
+}
+
+return Pair.of(copyHeader.substring(0, pos), copyHeader.substring(pos +
1));
Review comment:
I think this should use header instead of copyHeader
`Pair.of(header.substring(0, pos), header.substring(pos + 1))`? I think we can
just use a single variable here for clarity?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] lokeshj1703 commented on a change in pull request #1278: HDDS-1950. S3 MPU part-list call fails if there are no parts
lokeshj1703 commented on a change in pull request #1278: HDDS-1950. S3 MPU
part-list call fails if there are no parts
URL: https://github.com/apache/hadoop/pull/1278#discussion_r313386459
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -1329,8 +1329,16 @@ public OmMultipartUploadListParts listParts(String
volumeName,
multipartKeyInfo.getPartKeyInfoMap();
Iterator> partKeyInfoMapIterator =
partKeyInfoMap.entrySet().iterator();
-HddsProtos.ReplicationType replicationType =
-partKeyInfoMap.firstEntry().getValue().getPartKeyInfo().getType();
+
+OmKeyInfo omKeyInfo =
+metadataManager.getOpenKeyTable().get(multipartKey);
+
+if (omKeyInfo == null) {
+ throw new IllegalStateException(
+ "Open key is missing for multipart upload " + multipartKey);
+}
+
+HddsProtos.ReplicationType replicationType = omKeyInfo.getType();
Review comment:
We are setting replicationType twice- line 1358 and 1341. Can we add this
openKeyTable get op inside an if condition so that it is executed only if no
parts are present in the key?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] lokeshj1703 commented on a change in pull request #1278: HDDS-1950. S3 MPU part-list call fails if there are no parts
lokeshj1703 commented on a change in pull request #1278: HDDS-1950. S3 MPU
part-list call fails if there are no parts
URL: https://github.com/apache/hadoop/pull/1278#discussion_r313388730
##
File path:
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerUnit.java
##
@@ -0,0 +1,93 @@
+package org.apache.hadoop.ozone.om;
+
+import java.io.IOException;
+import java.util.ArrayList;
+
+import org.apache.hadoop.hdds.HddsConfigKeys;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationType;
+import org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyArgs;
+import org.apache.hadoop.ozone.om.helpers.OmKeyArgs.Builder;
+import org.apache.hadoop.ozone.om.helpers.OmMultipartInfo;
+import org.apache.hadoop.ozone.om.helpers.OmMultipartUploadListParts;
+import org.apache.hadoop.ozone.security.OzoneBlockTokenSecretManager;
+import org.apache.hadoop.test.GenericTestUtils;
+
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+/**
+ * Unit test key manager.
+ */
+public class TestKeyManagerUnit {
Review comment:
We already have a class for KeyManagerImpl unit tests "TestKeyManagerImpl".
I think we can use that or make this one specific for the s3 apis?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1226: HDDS-1610. applyTransaction failure should not be lost on restart.
hadoop-yetus commented on issue #1226: HDDS-1610. applyTransaction failure should not be lost on restart. URL: https://github.com/apache/hadoop/pull/1226#issuecomment-520836222 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 142 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 88 | Maven dependency ordering for branch | | +1 | mvninstall | 804 | trunk passed | | +1 | compile | 444 | trunk passed | | +1 | checkstyle | 86 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 958 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 166 | trunk passed | | 0 | spotbugs | 445 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 644 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 32 | Maven dependency ordering for patch | | +1 | mvninstall | 595 | the patch passed | | +1 | compile | 401 | the patch passed | | +1 | cc | 401 | the patch passed | | +1 | javac | 401 | the patch passed | | -0 | checkstyle | 42 | hadoop-ozone: The patch generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 731 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 175 | the patch passed | | +1 | findbugs | 699 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 361 | hadoop-hdds in the patch passed. | | -1 | unit | 2308 | hadoop-ozone in the patch failed. | | +1 | asflicense | 45 | The patch does not generate ASF License warnings. | | | | 8928 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestContainerStateMachineFailures | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.container.server.TestSecureContainerServer | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | | hadoop.ozone.container.common.transport.server.ratis.TestCSMMetrics | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1226/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1226 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 07f043e4650b 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 0b507d2 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1226/4/artifact/out/diff-checkstyle-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1226/4/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1226/4/testReport/ | | Max. process+thread count | 4545 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/container-service hadoop-ozone/integration-test hadoop-ozone/tools U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1226/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 closed pull request #1284: HDDS-1952. Disable TestMiniChaosOzoneCluster in integration.sh
nandakumar131 closed pull request #1284: HDDS-1952. Disable TestMiniChaosOzoneCluster in integration.sh URL: https://github.com/apache/hadoop/pull/1284 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1284: HDDS-1952. Disable TestMiniChaosOzoneCluster in integration.sh
adoroszlai commented on issue #1284: HDDS-1952. Disable TestMiniChaosOzoneCluster in integration.sh URL: https://github.com/apache/hadoop/pull/1284#issuecomment-520839636 Thank you @nandakumar131 for reviewing and merging it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16416) mark DynamoDBMetadataStore.deleteTrackingValueMap as final
[
https://issues.apache.org/jira/browse/HADOOP-16416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kevin su updated HADOOP-16416:
--
Status: Patch Available (was: Open)
> mark DynamoDBMetadataStore.deleteTrackingValueMap as final
> --
>
> Key: HADOOP-16416
> URL: https://issues.apache.org/jira/browse/HADOOP-16416
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: kevin su
>Priority: Trivial
> Attachments: HADOOP-16416.001.patch
>
>
> S3Guard's {{DynamoDBMetadataStore.deleteTrackingValueMap}} field is static
> and can/should be marked as final; its name changed to upper case to match
> the coding conventions.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on issue #1283: HDDS-1954. StackOverflowError in OzoneClientInvocationHandler
adoroszlai commented on issue #1283: HDDS-1954. StackOverflowError in OzoneClientInvocationHandler URL: https://github.com/apache/hadoop/pull/1283#issuecomment-520848774 @anuengineer @dineshchitlangia please review when you have some time This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1234: HDDS-1917. TestOzoneRpcClientAbstract is failing.
hadoop-yetus commented on issue #1234: HDDS-1917. TestOzoneRpcClientAbstract is failing. URL: https://github.com/apache/hadoop/pull/1234#issuecomment-520859188 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 129 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 36 | Maven dependency ordering for branch | | +1 | mvninstall | 833 | trunk passed | | -1 | compile | 262 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 120 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 1107 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 216 | trunk passed | | 0 | spotbugs | 509 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 758 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for patch | | +1 | mvninstall | 670 | the patch passed | | +1 | compile | 442 | the patch passed | | -1 | javac | 322 | hadoop-ozone generated 4 new + 4 unchanged - 0 fixed = 8 total (was 4) | | -0 | checkstyle | 49 | hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 748 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 182 | the patch passed | | +1 | findbugs | 753 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 364 | hadoop-hdds in the patch passed. | | -1 | unit | 2564 | hadoop-ozone in the patch failed. | | +1 | asflicense | 45 | The patch does not generate ASF License warnings. | | | | 9599 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestFailureHandlingByClient | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1234 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 2d1d2eed053a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 0b507d2 | | Default Java | 1.8.0_212 | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/7/artifact/out/branch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/7/artifact/out/diff-compile-javac-hadoop-ozone.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/7/artifact/out/diff-checkstyle-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/7/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/7/testReport/ | | Max. process+thread count | 5065 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/7/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy
bgaborg commented on a change in pull request #1229: HADOOP-16490. Improve
S3Guard handling of FNFEs in copy
URL: https://github.com/apache/hadoop/pull/1229#discussion_r313424521
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java
##
@@ -80,13 +82,18 @@
@SuppressWarnings("visibilitymodifier") // I want a struct of finals, for
real.
public class S3ARetryPolicy implements RetryPolicy {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ S3ARetryPolicy.class);
+
+ private final Configuration configuration;
+
/** Final retry policy we end up with. */
private final RetryPolicy retryPolicy;
// Retry policies for mapping exceptions to
/** Base policy from configuration. */
- protected final RetryPolicy fixedRetries;
+ protected final RetryPolicy defaultRetries;
Review comment:
maybe exponentialBackoffRetry would be a better name for this, or do you
plan to make the retry implementation pluggable here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy
bgaborg commented on a change in pull request #1229: HADOOP-16490. Improve
S3Guard handling of FNFEs in copy
URL: https://github.com/apache/hadoop/pull/1229#discussion_r313422737
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -2603,6 +2607,30 @@ S3AFileStatus innerGetFileStatus(final Path f,
entryPoint(INVOCATION_GET_FILE_STATUS);
checkNotClosed();
final Path path = qualify(f);
+return resolveFileStatus(path, needEmptyDirectoryFlag, false);
+ }
+
+
+ /**
+ * Get the status of a file or directory, first through S3Guard and then
+ * through S3.
+ * The S3 probes can leave 404 responses in the S3 load balancers; if
+ * a check is only needed for a directory, declaring this saves time and
+ * avoids creating one for the object.
+ * When only probing for directories, if an entry for a file is found in
+ * S3Guard it is returned, but checks for updated values are skipped.
+ * @param path fully qualified path
+ * @param needEmptyDirectoryFlag if true, implementation will calculate
+ *a TRUE or FALSE value for {@link S3AFileStatus#isEmptyDirectory()}
+ * @param onlyProbeForDirectory skip the simple object probe
+ * @return a S3AFileStatus object
+ * @throws FileNotFoundException when the path does not exist
+ * @throws IOException on other problems.
+ */
+ private S3AFileStatus resolveFileStatus(final Path path,
Review comment:
I think if we plan to refactor something out from S3AFs it would be good to
start now :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy
bgaborg commented on a change in pull request #1229: HADOOP-16490. Improve
S3Guard handling of FNFEs in copy
URL: https://github.com/apache/hadoop/pull/1229#discussion_r313423481
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -3177,10 +3212,29 @@ private CopyResult copyFile(String srcKey, String
dstKey, long size,
String action = "copyFile(" + srcKey + ", " + dstKey + ")";
Invoker readInvoker = readContext.getReadInvoker();
-ObjectMetadata srcom =
-once(action, srcKey,
-() ->
-getObjectMetadata(srcKey, changeTracker, readInvoker, "copy"));
+ObjectMetadata srcom;
+try {
+ srcom = once(action, srcKey,
+ () ->
+ getObjectMetadata(srcKey, changeTracker, readInvoker, "copy"));
+} catch (FileNotFoundException e) {
+ // if rename fails at this point it means that the expected file was not
+ // found.
+ // The cause is believed to always be one of
+ // - File has been deleted since LIST/S3Guard list knew of it.
+ // - S3Guard is asking for a specific version and it's been removed by
+ //lifecycle rules.
+ // - there's a 404 cached in S3 load balancers
+ // We create an exception, but the text depends on the S3Guard state
+ String message = hasMetadataStore()
Review comment:
good idea to have detailed debug here!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy
bgaborg commented on a change in pull request #1229: HADOOP-16490. Improve
S3Guard handling of FNFEs in copy
URL: https://github.com/apache/hadoop/pull/1229#discussion_r313421971
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##
@@ -2603,6 +2607,30 @@ S3AFileStatus innerGetFileStatus(final Path f,
entryPoint(INVOCATION_GET_FILE_STATUS);
checkNotClosed();
final Path path = qualify(f);
+return resolveFileStatus(path, needEmptyDirectoryFlag, false);
+ }
+
+
+ /**
+ * Get the status of a file or directory, first through S3Guard and then
+ * through S3.
+ * The S3 probes can leave 404 responses in the S3 load balancers; if
+ * a check is only needed for a directory, declaring this saves time and
+ * avoids creating one for the object.
+ * When only probing for directories, if an entry for a file is found in
+ * S3Guard it is returned, but checks for updated values are skipped.
+ * @param path fully qualified path
+ * @param needEmptyDirectoryFlag if true, implementation will calculate
+ *a TRUE or FALSE value for {@link S3AFileStatus#isEmptyDirectory()}
+ * @param onlyProbeForDirectory skip the simple object probe
+ * @return a S3AFileStatus object
+ * @throws FileNotFoundException when the path does not exist
+ * @throws IOException on other problems.
+ */
+ private S3AFileStatus resolveFileStatus(final Path path,
Review comment:
Do you plan to refactor this to a class - like S3AFileStatusResolver? What's
the reasoning behind factoring this out from innerGetFileStatus? This leaves
that method just a call for this method - plus a few other things, but
basically, it's just that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy
bgaborg commented on a change in pull request #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy URL: https://github.com/apache/hadoop/pull/1229#discussion_r313354887 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/RemoteFileChangedException.java ## @@ -35,6 +35,20 @@ public static final String PRECONDITIONS_FAILED = "Constraints of request were unsatisfiable"; + /** + * While trying to get information on a file known to S3Guard, the + * file never became visible in S3. + */ + public static final String FILE_NEVER_FOUND = + "File to rename not found on S3 after repeated attempts"; Review comment: maybe add that this has happened with a guarded s3 store. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16416) mark DynamoDBMetadataStore.deleteTrackingValueMap as final
[
https://issues.apache.org/jira/browse/HADOOP-16416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906284#comment-16906284
]
Hadoop QA commented on HADOOP-16416:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
3s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
18s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HADOOP-16416 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12974667/HADOOP-16416.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2bea91547c8b 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3
08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3dc22d6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16475/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16475/testReport/ |
| Max. process+thread count | 337 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/Pre
[jira] [Commented] (HADOOP-16264) [JDK11] Track failing Hadoop unit tests
[ https://issues.apache.org/jira/browse/HADOOP-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906297#comment-16906297 ] Adam Antal commented on HADOOP-16264: - I also add my findings for the hadoop-yarn-project: [^test-run-openjdk11.0.4-macos10.14.5.txt]. > [JDK11] Track failing Hadoop unit tests > --- > > Key: HADOOP-16264 > URL: https://issues.apache.org/jira/browse/HADOOP-16264 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.1.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: test-run-openjdk11.0.4-macos10.14.5.txt, test-run1.tgz, > test-run2.log, test-run2.log.gz, test-run3-openjdk11.0.3u7-ubuntu19.04.log > > > Although there are still a lot of work before we could compile Hadoop with > JDK 11 (HADOOP-15338), it is possible to compile Hadoop with JDK 8 and run > (e.g. HDFS NN/DN,YARN NM/RM) on JDK 11 at this moment. > But after compiling branch-3.1.2 with JDK 8, I ran unit tests with JDK 11 and > there are a LOT of unit test failures (44 out of 96 maven projects contain at > least one unit test failures according to maven reactor summary). This may > well indicate some functionalities are actually broken on JDK 11. Some of > them already have a jira number. Some of them might have been fixed in 3.2.0. > Some of them might share the same root cause. > By definition, this jira should be part of HADOOP-15338. But the goal of this > one is just to keep track of unit test failures and (hopefully) resolve all > of them soon. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16264) [JDK11] Track failing Hadoop unit tests
[ https://issues.apache.org/jira/browse/HADOOP-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated HADOOP-16264: Attachment: test-run-openjdk11.0.4-macos10.14.5.txt > [JDK11] Track failing Hadoop unit tests > --- > > Key: HADOOP-16264 > URL: https://issues.apache.org/jira/browse/HADOOP-16264 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.1.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: test-run-openjdk11.0.4-macos10.14.5.txt, test-run1.tgz, > test-run2.log, test-run2.log.gz, test-run3-openjdk11.0.3u7-ubuntu19.04.log > > > Although there are still a lot of work before we could compile Hadoop with > JDK 11 (HADOOP-15338), it is possible to compile Hadoop with JDK 8 and run > (e.g. HDFS NN/DN,YARN NM/RM) on JDK 11 at this moment. > But after compiling branch-3.1.2 with JDK 8, I ran unit tests with JDK 11 and > there are a LOT of unit test failures (44 out of 96 maven projects contain at > least one unit test failures according to maven reactor summary). This may > well indicate some functionalities are actually broken on JDK 11. Some of > them already have a jira number. Some of them might have been fixed in 3.2.0. > Some of them might share the same root cause. > By definition, this jira should be part of HADOOP-15338. But the goal of this > one is just to keep track of unit test failures and (hopefully) resolve all > of them soon. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16503) [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
[
https://issues.apache.org/jira/browse/HADOOP-16503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906301#comment-16906301
]
Adam Antal commented on HADOOP-16503:
-
Verification failed, I still managed to see the exception after applying the
patch. Will look into this a bit further later.
> [JDK11] TestLeafQueue tests are failing due to WrongTypeOfReturnValue
> -
>
> Key: HADOOP-16503
> URL: https://issues.apache.org/jira/browse/HADOOP-16503
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16503.001.patch
>
>
> Many of the tests in
> {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue}}
> fails with the following error message running on JDK11:
> {noformat}
> [ERROR]
> testSingleQueueWithOneUser(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue)
> Time elapsed: 0.204 s <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
> YarnConfiguration cannot be returned by getRMNodes()
> getRMNodes() should return ConcurrentMap
> ***
> If you're unsure why you're getting above error read on.
> Due to the nature of the syntax above problem might occur because:
> 1. This exception *might* occur in wrongly written multi-threaded tests.
>Please refer to Mockito FAQ on limitations of concurrency testing.
> 2. A spy is stubbed using when(spy.foo()).then() syntax. It is safer to stub
> spies -
>- with doReturn|Throw() family of methods. More in javadocs for
> Mockito.spy() method.
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUpInternal(TestLeafQueue.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.setUp(TestLeafQueue.java:144)
>...
> {noformat}
> This is due to the actual execution of the call, while we need to record only
> the invocation of it. According to the javadocs and other folks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukul1987 commented on a change in pull request #1287: HDDS-1956. Aged IO Thread exits on first read
mukul1987 commented on a change in pull request #1287: HDDS-1956. Aged IO
Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313465776
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##
@@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
String keyName = null;
try {
if (agedWriteProbability.isTrue()) {
- keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+ keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
Review comment:
Aged file index always start with 0, so we will never read the first file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukul1987 commented on a change in pull request #1287: HDDS-1956. Aged IO Thread exits on first read
mukul1987 commented on a change in pull request #1287: HDDS-1956. Aged IO
Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313464607
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##
@@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
String keyName = null;
try {
if (agedWriteProbability.isTrue()) {
- keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+ keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
agedLoadBucket, threadName);
} else {
- keyName = getKeyToRead();
- if (keyName != null) {
+ Optional index = randomKeyToRead();
+ if (index.isPresent()) {
+keyName = getKeyName(index.get(), threadName);
readData(agedLoadBucket, keyName);
Review comment:
lets pass the index to readData, so that we can eliminate the split function
inside read.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on a change in pull request #1287: HDDS-1956. Aged IO Thread exits on first read
adoroszlai commented on a change in pull request #1287: HDDS-1956. Aged IO
Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313467570
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##
@@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
String keyName = null;
try {
if (agedWriteProbability.isTrue()) {
- keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+ keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
agedLoadBucket, threadName);
} else {
- keyName = getKeyToRead();
- if (keyName != null) {
+ Optional index = randomKeyToRead();
+ if (index.isPresent()) {
+keyName = getKeyName(index.get(), threadName);
readData(agedLoadBucket, keyName);
Review comment:
OK, makes sense.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on a change in pull request #1287: HDDS-1956. Aged IO Thread exits on first read
adoroszlai commented on a change in pull request #1287: HDDS-1956. Aged IO
Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313470272
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##
@@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
String keyName = null;
try {
if (agedWriteProbability.isTrue()) {
- keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+ keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
Review comment:
Can you please clarify what's the issue here?
Random index selection gives a number in `[0, currentIndex)` (see
`RandomUtils.nextInt(0, currentIndex)`).
Previously files were written starting at 1 and `currentIndex` equaled the
last written key index. Now `currentIndex` is last written key index + 1, so
the random choice always produces an existing key index.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve fsimage load time by writing sub-sections to the fsimage index
sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve
fsimage load time by writing sub-sections to the fsimage index
URL: https://github.com/apache/hadoop/pull/1028#discussion_r313481427
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
##
@@ -525,6 +689,59 @@ long save(File file, FSImageCompression compression)
throws IOException {
}
}
+private void enableSubSectionsIfRequired() {
+ boolean parallelEnabled = conf.getBoolean(
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY,
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_DEFAULT);
+ int inodeThreshold = conf.getInt(
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY,
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_DEFAULT);
+ int targetSections = conf.getInt(
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY,
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_DEFAULT);
+ boolean compressionEnabled = conf.getBoolean(
+ DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY,
+ DFSConfigKeys.DFS_IMAGE_COMPRESS_DEFAULT);
+
+ if (parallelEnabled) {
+if (compressionEnabled) {
+ LOG.warn("Parallel Image loading is not supported when {} is set to"
+
+ " true. Parallel loading will be disabled.",
Review comment:
I have added a note to hdfs-default.xml for this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve fsimage load time by writing sub-sections to the fsimage index
sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve
fsimage load time by writing sub-sections to the fsimage index
URL: https://github.com/apache/hadoop/pull/1028#discussion_r313482053
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
##
@@ -525,6 +689,59 @@ long save(File file, FSImageCompression compression)
throws IOException {
}
}
+private void enableSubSectionsIfRequired() {
+ boolean parallelEnabled = conf.getBoolean(
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY,
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_DEFAULT);
+ int inodeThreshold = conf.getInt(
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY,
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_DEFAULT);
+ int targetSections = conf.getInt(
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY,
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_DEFAULT);
+ boolean compressionEnabled = conf.getBoolean(
+ DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY,
+ DFSConfigKeys.DFS_IMAGE_COMPRESS_DEFAULT);
+
+ if (parallelEnabled) {
+if (compressionEnabled) {
+ LOG.warn("Parallel Image loading is not supported when {} is set to"
+
+ " true. Parallel loading will be disabled.",
+ DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY);
+ writeSubSections = false;
+ return;
+}
+if (targetSections <= 0) {
+ LOG.warn("{} is set to {}. It must be greater than zero. Setting to"
+
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16448) Connection to Hadoop homepage is not secure
[ https://issues.apache.org/jira/browse/HADOOP-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906349#comment-16906349 ] Sandeep Nemuri commented on HADOOP-16448: - Apart from the reported urls, updated all the possible ones from http to https. Kindly review. > Connection to Hadoop homepage is not secure > --- > > Key: HADOOP-16448 > URL: https://issues.apache.org/jira/browse/HADOOP-16448 > Project: Hadoop Common > Issue Type: Bug > Components: website >Affects Versions: 0.23.11, 2.4.0, 2.5.0, 2.4.1, 2.5.1, 2.5.2, 2.6.0, > 2.6.1, 2.7.0, 2.8.0, 2.7.1, 2.7.2, 2.6.2, 2.6.3, 2.7.3, 2.9.0, 2.6.4, 2.6.5, > 2.7.4, 2.8.1, 2.8.2, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.9.1, 3.0.1, 2.8.4, 2.7.6, > 3.2.0, 3.0.2, 3.1.1, 2.9.2, 3.0.3, 2.7.7, 2.8.5 >Reporter: Kaspar Tint >Assignee: Sandeep Nemuri >Priority: Major > Labels: newbie > Attachments: Screen Shot 2019-07-23 at 9.37.54 AM.png > > > When visiting the [Hadoop > website|https://hadoop.apache.org/docs/r3.2.0/index.html] with the latest > Firefox browser (v 68.0.1) it appears that the website cannot be reached > through secure means by default. > The culprit seems to be the fact that the two header images presented on the > page are loaded in via *HTTP* > !Screen Shot 2019-07-23 at 9.37.54 AM.png!. > These images are located in the respective locations: > http://hadoop.apache.org/images/hadoop-logo.jpg > http://www.apache.org/images/asf_logo_wide.png > These images can be reached also from the following locations: > https://hadoop.apache.org/images/hadoop-logo.jpg > https://www.apache.org/images/asf_logo_wide.png > As one can see, a fix could be made to use a more safe way of including in > the two header pictures to the page. > I feel like I am in danger when reading the Hadoop documentation from the > official Hadoop webpage in a non secure way. Thus I felt the need to open > this ticket and raise the issue in order to have a future where everyone can > learn from Hadoop documentation in a safe and secure way. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukul1987 commented on a change in pull request #1287: HDDS-1956. Aged IO Thread exits on first read
mukul1987 commented on a change in pull request #1287: HDDS-1956. Aged IO
Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313487361
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##
@@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
String keyName = null;
try {
if (agedWriteProbability.isTrue()) {
- keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+ keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
Review comment:
With getAndIncrement, the first key is written with index 0.
in randomKeyToRead, for current index of 0, we return the Optional.empty().
And hence the first key will never be read.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1234: HDDS-1917. TestOzoneRpcClientAbstract is failing.
hadoop-yetus commented on issue #1234: HDDS-1917. TestOzoneRpcClientAbstract is failing. URL: https://github.com/apache/hadoop/pull/1234#issuecomment-520903069 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 71 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 42 | Maven dependency ordering for branch | | +1 | mvninstall | 625 | trunk passed | | +1 | compile | 368 | trunk passed | | +1 | checkstyle | 72 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 951 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 171 | trunk passed | | 0 | spotbugs | 447 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 657 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 20 | Maven dependency ordering for patch | | +1 | mvninstall | 567 | the patch passed | | +1 | compile | 385 | the patch passed | | +1 | javac | 385 | the patch passed | | +1 | checkstyle | 84 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 775 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 164 | the patch passed | | +1 | findbugs | 668 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 362 | hadoop-hdds in the patch passed. | | -1 | unit | 2532 | hadoop-ozone in the patch failed. | | +1 | asflicense | 159 | The patch does not generate ASF License warnings. | | | | 8853 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.pipeline.TestPipelineClose | | | hadoop.hdds.scm.pipeline.TestSCMPipelineManager | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1234 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 70e674f21ba1 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 3dc22d6 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/8/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/8/testReport/ | | Max. process+thread count | 4371 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1234/8/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve fsimage load time by writing sub-sections to the fsimage index
sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve
fsimage load time by writing sub-sections to the fsimage index
URL: https://github.com/apache/hadoop/pull/1028#discussion_r313491763
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
##
@@ -294,6 +368,19 @@ public int compare(FileSummary.Section s1,
FileSummary.Section s2) {
* a particular step to be started for once.
*/
Step currentStep = null;
+ boolean loadInParallel =
+ conf.getBoolean(DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY,
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_DEFAULT);
+ // TODO - check for compression and if enabled disable parallel
Review comment:
I have addressed this TODO and also refactored the code a little to allow
reuse between the saving and loading sections.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve fsimage load time by writing sub-sections to the fsimage index
sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve
fsimage load time by writing sub-sections to the fsimage index
URL: https://github.com/apache/hadoop/pull/1028#discussion_r313493210
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
##
@@ -250,6 +257,73 @@ void load(File file) throws IOException {
}
}
+/**
+ * Given a FSImage FileSummary.section, return a LimitInput stream set to
+ * the starting position of the section and limited to the section length.
+ * @param section The FileSummary.Section containing the offset and length
+ * @param compressionCodec The compression codec in use, if any
+ * @return An InputStream for the given section
+ * @throws IOException
+ */
+public InputStream getInputStreamForSection(FileSummary.Section section,
+String compressionCodec)
+throws IOException {
+ FileInputStream fin = new FileInputStream(filename);
+ FileChannel channel = fin.getChannel();
+ channel.position(section.getOffset());
+ InputStream in = new BufferedInputStream(new LimitInputStream(fin,
+ section.getLength()));
+
+ in = FSImageUtil.wrapInputStreamForCompression(conf,
+ compressionCodec, in);
+ return in;
+}
+
+/**
+ * Takes an ArrayList of Section's and removes all Section's whose
+ * name ends in _SUB, indicating they are sub-sections. The original
+ * array list is modified and a new list of the removed Section's is
+ * returned.
+ * @param sections Array List containing all Sections and Sub Sections
+ * in the image.
+ * @return ArrayList of the sections removed, or an empty list if none are
+ * removed.
+ */
+private ArrayList getAndRemoveSubSections(
+ArrayList sections) {
+ ArrayList subSections = new ArrayList<>();
+ Iterator iter = sections.iterator();
+ while (iter.hasNext()) {
+FileSummary.Section s = iter.next();
+String name = s.getName();
+if (name.matches(".*_SUB$")) {
+ subSections.add(s);
+ iter.remove();
+}
+ }
+ return subSections;
+}
+
+/**
+ * Given an ArrayList of Section's, return all Section's with the given
+ * name, or an empty list if none are found.
+ * @param sections ArrayList of the Section's to search though
+ * @param name The name of the Sections to search for
+ * @return ArrayList of the sections matching the given name
+ */
+private ArrayList getSubSectionsOfName(
+ArrayList sections, SectionName name) {
+ ArrayList subSec = new ArrayList<>();
+ for (FileSummary.Section s : sections) {
+String n = s.getName();
+SectionName sectionName = SectionName.fromString(n);
+if (sectionName == name) {
+ subSec.add(s);
+}
+ }
+ return subSec;
+}
+
Review comment:
It was a long method already, and this has made it longer. I will see if I
can find a way to refactor it without introducing too much change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on a change in pull request #1287: HDDS-1956. Aged IO Thread exits on first read
adoroszlai commented on a change in pull request #1287: HDDS-1956. Aged IO
Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313494917
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##
@@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
String keyName = null;
try {
if (agedWriteProbability.isTrue()) {
- keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+ keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
Review comment:
Before key with index=0 is written, the current index is incremented to 1.
Please see my [example
log](https://github.com/apache/hadoop/pull/1287#issue-306801835) for reads of
the first key right after the first write (and even after the second one).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on a change in pull request #1287: HDDS-1956. Aged IO Thread exits on first read
adoroszlai commented on a change in pull request #1287: HDDS-1956. Aged IO
Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313494917
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##
@@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
String keyName = null;
try {
if (agedWriteProbability.isTrue()) {
- keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+ keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
Review comment:
Before key with index=0 is written, the current index is incremented to 1.
So `randomKeyToRead` only sees `currentIndex==0` if there is no key yet.
Please see my [example
log](https://github.com/apache/hadoop/pull/1287#issue-306801835) for reads of
the first key right after the first write (and even after the second one).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HADOOP-16459: - Fix Version/s: 3.1.3 3.2.1 3.0.4 > Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the > RPC layer" to branch-2 > > > Key: HADOOP-16459 > URL: https://issues.apache.org/jira/browse/HADOOP-16459 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Fix For: 3.0.4, 3.2.1, 3.1.3 > > Attachments: HADOOP-16266-branch-2.000.patch, > HADOOP-16266-branch-2.001.patch, HADOOP-16266-branch-3.0.000.patch, > HADOOP-16266-branch-3.1.000.patch, HADOOP-16266-branch-3.2.000.patch > > > We would like to target pulling HADOOP-16266, an important operability > enhancement and prerequisite for HDFS-14403, into branch-2. > It's only present in trunk now so we also need to backport through the 3.x > lines. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukul1987 commented on a change in pull request #1287: HDDS-1956. Aged IO Thread exits on first read
mukul1987 commented on a change in pull request #1287: HDDS-1956. Aged IO
Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313496805
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##
@@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
String keyName = null;
try {
if (agedWriteProbability.isTrue()) {
- keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+ keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
Review comment:
got that, Thanks for the explanation. This comment can be ignored.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 opened a new pull request #1288: HDDS-1961. TestStorageContainerManager#testScmProcessDatanodeHeartbeat is flaky.
nandakumar131 opened a new pull request #1288: HDDS-1961. TestStorageContainerManager#testScmProcessDatanodeHeartbeat is flaky. URL: https://github.com/apache/hadoop/pull/1288 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on a change in pull request #1287: HDDS-1956. Aged IO Thread exits on first read
adoroszlai commented on a change in pull request #1287: HDDS-1956. Aged IO
Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313497795
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##
@@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
String keyName = null;
try {
if (agedWriteProbability.isTrue()) {
- keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+ keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
agedLoadBucket, threadName);
} else {
- keyName = getKeyToRead();
- if (keyName != null) {
+ Optional index = randomKeyToRead();
+ if (index.isPresent()) {
+keyName = getKeyName(index.get(), threadName);
readData(agedLoadBucket, keyName);
Review comment:
Thanks for the idea, pushed a commit with this change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] avijayanhwx commented on issue #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager
avijayanhwx commented on issue #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#issuecomment-520921949 cc @mukul1987 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HADOOP-16459: - Attachment: HADOOP-16266-branch-2.002.patch > Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the > RPC layer" to branch-2 > > > Key: HADOOP-16459 > URL: https://issues.apache.org/jira/browse/HADOOP-16459 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Fix For: 3.0.4, 3.2.1, 3.1.3 > > Attachments: HADOOP-16266-branch-2.000.patch, > HADOOP-16266-branch-2.001.patch, HADOOP-16266-branch-2.002.patch, > HADOOP-16266-branch-3.0.000.patch, HADOOP-16266-branch-3.1.000.patch, > HADOOP-16266-branch-3.2.000.patch > > > We would like to target pulling HADOOP-16266, an important operability > enhancement and prerequisite for HDFS-14403, into branch-2. > It's only present in trunk now so we also need to backport through the 3.x > lines. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906420#comment-16906420 ] Erik Krogen commented on HADOOP-16459: -- Re-visiting this branch-2 backport now that HDFS-14204 has been completed -- see [^HADOOP-16266-branch-2.002.patch]. I was able to use the branch-3.0 patch as a cherry-pick, and made the Java 7 compatibility fixes discussed in my [previous comment|https://issues.apache.org/jira/secure/EditComment!default.jspa?id=13246892&commentId=16892216]. [~vagarychen] or [~elgoiri], care to take another look? > Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the > RPC layer" to branch-2 > > > Key: HADOOP-16459 > URL: https://issues.apache.org/jira/browse/HADOOP-16459 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Fix For: 3.0.4, 3.2.1, 3.1.3 > > Attachments: HADOOP-16266-branch-2.000.patch, > HADOOP-16266-branch-2.001.patch, HADOOP-16266-branch-2.002.patch, > HADOOP-16266-branch-3.0.000.patch, HADOOP-16266-branch-3.1.000.patch, > HADOOP-16266-branch-3.2.000.patch > > > We would like to target pulling HADOOP-16266, an important operability > enhancement and prerequisite for HDFS-14403, into branch-2. > It's only present in trunk now so we also need to backport through the 3.x > lines. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16351) Change ":" to ApplicationConstants.CLASS_PATH_SEPARATOR
[
https://issues.apache.org/jira/browse/HADOOP-16351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906424#comment-16906424
]
Wei-Chiu Chuang commented on HADOOP-16351:
--
[~pingsutw] this looks good. Have you verified it on a Windows machine? Sadly I
don't have a Windows box to try it out.
> Change ":" to ApplicationConstants.CLASS_PATH_SEPARATOR
> ---
>
> Key: HADOOP-16351
> URL: https://issues.apache.org/jira/browse/HADOOP-16351
> Project: Hadoop Common
> Issue Type: Task
> Components: common
>Affects Versions: 3.1.2
>Reporter: kevin su
>Assignee: kevin su
>Priority: Trivial
> Fix For: 3.1.2
>
> Attachments: HADOOP-16351.01.patch
>
>
> under distributedshell/Clients.java
> We should change ":" to ApplicationConstants.CLASS_PATH_SEPARATOR, so it
> could also support Windows client
> {code}
> // add the runtime classpath needed for tests to work
> if (conf.getBoolean(YarnConfiguration.IS_MINI_YARN_CLUSTER, false)) {
> classPathEnv.append(':')
> .append(System.getProperty("java.class.path"));
> }
> {code}
> {code}
> // add the runtime classpath needed for tests to work
> if (conf.getBoolean(YarnConfiguration.IS_MINI_YARN_CLUSTER, false)) {
> classPathEnv.append(ApplicationConstants.CLASS_PATH_SEPARATOR)
> .append(System.getProperty("java.class.path"));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #1253: HDFS-8631. WebHDFS : Support setQuota
sunchao commented on a change in pull request #1253: HDFS-8631. WebHDFS :
Support setQuota
URL: https://github.com/apache/hadoop/pull/1253#discussion_r313517504
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/resources/TestParam.java
##
@@ -49,7 +51,7 @@
@Test
public void testAccessTimeParam() {
final AccessTimeParam p = new AccessTimeParam(AccessTimeParam.DEFAULT);
-Assert.assertEquals(-1L, p.getValue().longValue());
+assertEquals(-1L, p.getValue().longValue());
Review comment:
Sure. Will revert the unrelated changes in this class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] avijayanhwx commented on a change in pull request #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager
avijayanhwx commented on a change in pull request #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager URL: https://github.com/apache/hadoop/pull/1259#discussion_r313542436 ## File path: hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServerConfigKeys.java ## @@ -114,7 +114,7 @@ public static final String OZONE_RECON_TASK_THREAD_COUNT_KEY = "ozone.recon.task.thread.count"; - public static final int OZONE_RECON_TASK_THREAD_COUNT_DEFAULT = 1; + public static final int OZONE_RECON_TASK_THREAD_COUNT_DEFAULT = 5; Review comment: Sure, I can address this later when the number of tasks becomes sizable. Right now, we have only 2. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16513) Upgrade Bootstrap, jquery and wildfly
Vivek Ratnavel Subramanian created HADOOP-16513: --- Summary: Upgrade Bootstrap, jquery and wildfly Key: HADOOP-16513 URL: https://issues.apache.org/jira/browse/HADOOP-16513 Project: Hadoop Common Issue Type: Bug Components: website Reporter: Vivek Ratnavel Subramanian The current versions of bootstrap, jquery and wildfly have multiple medium severity CVEs reported till date and needs to be updated to the latest versions with no reported CVEs. I suggest updating the following libraries: ||Library||From version||To version|| |Bootstrap|3.3.7|3.4.1| |jQuery|3.3.1|3.4.1| |Wildfly|11.0.0.Beta1|12.0.0| -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16513) Upgrade Bootstrap, jquery and wildfly
[ https://issues.apache.org/jira/browse/HADOOP-16513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey reassigned HADOOP-16513: - Assignee: Vivek Ratnavel Subramanian > Upgrade Bootstrap, jquery and wildfly > - > > Key: HADOOP-16513 > URL: https://issues.apache.org/jira/browse/HADOOP-16513 > Project: Hadoop Common > Issue Type: Bug > Components: website >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > > The current versions of bootstrap, jquery and wildfly have multiple medium > severity CVEs reported till date and needs to be updated to the latest > versions with no reported CVEs. > > I suggest updating the following libraries: > ||Library||From version||To version|| > |Bootstrap|3.3.7|3.4.1| > |jQuery|3.3.1|3.4.1| > |Wildfly|11.0.0.Beta1|12.0.0| -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1287: HDDS-1956. Aged IO Thread exits on first read
hadoop-yetus commented on issue #1287: HDDS-1956. Aged IO Thread exits on first read URL: https://github.com/apache/hadoop/pull/1287#issuecomment-520953642 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 41 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 639 | trunk passed | | +1 | compile | 389 | trunk passed | | +1 | checkstyle | 66 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | -1 | shadedclient | 864 | branch has errors when building and testing our client artifacts. | | +1 | javadoc | 166 | trunk passed | | 0 | spotbugs | 465 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 670 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 567 | the patch passed | | +1 | compile | 374 | the patch passed | | +1 | javac | 374 | the patch passed | | +1 | checkstyle | 73 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 691 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 153 | the patch passed | | -1 | findbugs | 320 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 13 | hadoop-hdds in the patch failed. | | -1 | unit | 31 | hadoop-ozone in the patch failed. | | 0 | asflicense | 31 | ASF License check generated no output? | | | | 5461 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1287 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 512599488faa 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 2ac029b | | Default Java | 1.8.0_212 | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/2/artifact/out/patch-findbugs-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/2/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/2/testReport/ | | Max. process+thread count | 447 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/integration-test U: hadoop-ozone/integration-test | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[
https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906520#comment-16906520
]
Erik Krogen commented on HADOOP-15565:
--
Thanks for working on this [~LiJinglun]! Seems like a good problem to fix and I
think the idea is sound.
I didn't get a chance to look at the tests yet, but I have some comments from
an initial review of the production code:
* I don't think publicly exposing {{cacheSize()}} on {{FileSystem}} is a great
idea. Can we make it package-private, and if it is needed in non-package-local
tests, use a test utility to export it publicly?
* Is there a chance the cache will be accessed in a multi-threaded way? If so
we need to harden it for concurrent access. Looks like it will only work in a
single-threaded fashion currently. If the FS instances are actually all created
on startup, then I think we should explicitly populate the cache on startup.
* I agree that swallowing exceptions on child FS close is the right move, but
probably we should at least put them at INFO level?
* This seems less strict than {{FileSystem.CACHE}} when checking for equality;
it doesn't use the {{UserGroupInformation}} at all. I think this is safe
because the cache is local to a single {{ViewFileSystem}}, so all of the inner
cached instances must share the same UGI, but please help me to confirm.
* We can use {{Objects.hash()}} for the {{hashCode()}} method of {{Key}}.
* On {{ViewFileSystem}} L257, you shouldn't initialize {{fs}} -- you can just
declare it: {{FileSystem fs;}} (this allows the compiler to help ensure that
you remember to initialize it later)
> ViewFileSystem.close doesn't close child filesystems and causes FileSystem
> objects leak.
>
>
> Key: HADOOP-15565
> URL: https://issues.apache.org/jira/browse/HADOOP-15565
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HADOOP-15565.0001.patch, HADOOP-15565.0002.patch,
> HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, HADOOP-15565.0005.patch
>
>
> ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE.
> It's children filesystems are cached in FileSystem.CACHE and shared by all
> the ViewFileSystem instances. We could't simply close all the children
> filesystems because it will break the semantic of FileSystem.newInstance().
> We might add an inner cache to ViewFileSystem, let it cache all the children
> filesystems. The children filesystems are not shared any more. When
> ViewFileSystem is closed we close all the children filesystems in the inner
> cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't
> be too many FileSystem instances.
> The FileSystem.CACHE caches the ViewFileSysem instance and the other
> instances(the children filesystems) are cached in the inner cache.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1288: HDDS-1961. TestStorageContainerManager#testScmProcessDatanodeHeartbeat is flaky.
hadoop-yetus commented on issue #1288: HDDS-1961. TestStorageContainerManager#testScmProcessDatanodeHeartbeat is flaky. URL: https://github.com/apache/hadoop/pull/1288#issuecomment-520966378 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 43 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 618 | trunk passed | | +1 | compile | 374 | trunk passed | | +1 | checkstyle | 69 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 853 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 170 | trunk passed | | 0 | spotbugs | 428 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 621 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 569 | the patch passed | | +1 | compile | 384 | the patch passed | | +1 | javac | 384 | the patch passed | | +1 | checkstyle | 86 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 684 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 164 | the patch passed | | +1 | findbugs | 635 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 276 | hadoop-hdds in the patch passed. | | -1 | unit | 1753 | hadoop-ozone in the patch failed. | | +1 | asflicense | 45 | The patch does not generate ASF License warnings. | | | | 7535 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1288/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1288 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux d11f0f66d00d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 2ac029b | | Default Java | 1.8.0_222 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1288/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1288/1/testReport/ | | Max. process+thread count | 4842 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/integration-test U: hadoop-ozone/integration-test | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1288/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13868) New defaults for S3A multi-part configuration
[ https://issues.apache.org/jira/browse/HADOOP-13868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HADOOP-13868: --- Resolution: Fixed Status: Resolved (was: Patch Available) > New defaults for S3A multi-part configuration > - > > Key: HADOOP-13868 > URL: https://issues.apache.org/jira/browse/HADOOP-13868 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.7.0, 3.0.0-alpha1 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Attachments: HADOOP-13868.001.patch, HADOOP-13868.002.patch, > optimizing-multipart-s3a.sh > > > I've been looking at a big performance regression when writing to S3 from > Spark that appears to have been introduced with HADOOP-12891. > In the Amazon SDK, the default threshold for multi-part copies is 320x the > threshold for multi-part uploads (and the block size is 20x bigger), so I > don't think it's necessarily wise for us to have them be the same. > I did some quick tests and it seems to me the sweet spot when multi-part > copies start being faster is around 512MB. It wasn't as significant, but > using 104857600 (Amazon's default) for the blocksize was also slightly better. > I propose we do the following, although they're independent decisions: > (1) Split the configuration. Ideally, I'd like to have > fs.s3a.multipart.copy.threshold and fs.s3a.multipart.upload.threshold (and > corresponding properties for the block size). But then there's the question > of what to do with the existing fs.s3a.multipart.* properties. Deprecation? > Leave it as a short-hand for configuring both (that's overridden by the more > specific properties?). > (2) Consider increasing the default values. In my tests, 256 MB seemed to be > where multipart uploads came into their own, and 512 MB was where multipart > copies started outperforming the alternative. Would be interested to hear > what other people have seen. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] avijayanhwx commented on a change in pull request #1286: Add filter to scmcli listPipelines.
avijayanhwx commented on a change in pull request #1286: Add filter to scmcli
listPipelines.
URL: https://github.com/apache/hadoop/pull/1286#discussion_r313576403
##
File path:
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java
##
@@ -38,11 +38,34 @@
@CommandLine.ParentCommand
private SCMCLI parent;
+ @CommandLine.Option( names = {"-ff", "--filterByFactor"},
+ description = "Filter listed pipelines by Factor(ONE/one)",
defaultValue = "",
+ required = false)
+ private String factor;
+
+ @CommandLine.Option( names = {"-fs", "--filterByState"},
Review comment:
Nit. Maybe some other shorthand than "-fs", since it usually means
filesystem in HDFS world.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] avijayanhwx commented on issue #1283: HDDS-1954. StackOverflowError in OzoneClientInvocationHandler
avijayanhwx commented on issue #1283: HDDS-1954. StackOverflowError in OzoneClientInvocationHandler URL: https://github.com/apache/hadoop/pull/1283#issuecomment-520978220 LGTM +1 (non binding) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] dineshchitlangia commented on issue #1283: HDDS-1954. StackOverflowError in OzoneClientInvocationHandler
dineshchitlangia commented on issue #1283: HDDS-1954. StackOverflowError in OzoneClientInvocationHandler URL: https://github.com/apache/hadoop/pull/1283#issuecomment-520979717 LGTM +1 (non binding), also I do not think the test failures are related. I have seen those tests fail of late in all the PRs. I believe there are jiras logged already to fix them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] avijayanhwx commented on issue #1255: HDDS-1935. Improve the visibility with Ozone Insight tool
avijayanhwx commented on issue #1255: HDDS-1935. Improve the visibility with Ozone Insight tool URL: https://github.com/apache/hadoop/pull/1255#issuecomment-520979879 This is a very useful addition @elek. Is there any documentation or slides that I can look at to understand this more? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] dineshchitlangia commented on a change in pull request #1205: HDDS-1886. Use ArrayList#clear to address audit failure scenario
dineshchitlangia commented on a change in pull request #1205: HDDS-1886. Use
ArrayList#clear to address audit failure scenario
URL: https://github.com/apache/hadoop/pull/1205#discussion_r313579214
##
File path:
hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/TestOzoneAuditLogger.java
##
@@ -153,7 +153,7 @@ private void verifyLog(String expected) throws IOException
{
assertTrue(lines.size() != 0);
assertTrue(expected.equalsIgnoreCase(lines.get(0)));
//empty the file
-lines.remove(0);
+lines.clear();
Review comment:
@adoroszlai Could please you let me know if there are any other concerns
based on my response above and follow up jira? Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1204: HDDS-1768. Audit xxxAcl methods in OzoneManager
bharatviswa504 commented on a change in pull request #1204: HDDS-1768. Audit
xxxAcl methods in OzoneManager
URL: https://github.com/apache/hadoop/pull/1204#discussion_r313582727
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientForAclAuditLog.java
##
@@ -0,0 +1,422 @@
+package org.apache.hadoop.ozone.client.rpc;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.RandomStringUtils;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.OzoneAcl;
+import org.apache.hadoop.ozone.audit.AuditEventStatus;
+import org.apache.hadoop.ozone.audit.OMAction;
+import org.apache.hadoop.ozone.client.OzoneVolume;
+import org.apache.hadoop.ozone.security.acl.IAccessAuthorizer;
+import org.apache.hadoop.ozone.security.acl.OzoneObj;
+import org.apache.hadoop.ozone.security.acl.OzoneObjInfo;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.FixMethodOrder;
+import org.junit.Test;
+import org.junit.runners.MethodSorters;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneAcl.AclScope.ACCESS;
+import static
org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_ACL_AUTHORIZER_CLASS;
+import static
org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_ACL_AUTHORIZER_CLASS_NATIVE;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_ACL_ENABLED;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_ADMINISTRATORS;
+import static
org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_ADMINISTRATORS_WILDCARD;
+import static
org.apache.hadoop.ozone.security.acl.OzoneObj.ResourceType.VOLUME;
+import static org.apache.hadoop.ozone.security.acl.OzoneObj.StoreType.OZONE;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * This class is to test audit logs for xxxACL APIs of Ozone Client.
+ */
+@FixMethodOrder(MethodSorters.NAME_ASCENDING)
+public class TestOzoneRpcClientForAclAuditLog extends
+TestOzoneRpcClientAbstract {
+
+ private static UserGroupInformation ugi;
+ private static final OzoneAcl USER_ACL =
+ new OzoneAcl(IAccessAuthorizer.ACLIdentityType.USER,
Review comment:
Ya then I think making a new test class will be a good idea here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adoroszlai commented on a change in pull request #1205: HDDS-1886. Use ArrayList#clear to address audit failure scenario
adoroszlai commented on a change in pull request #1205: HDDS-1886. Use
ArrayList#clear to address audit failure scenario
URL: https://github.com/apache/hadoop/pull/1205#discussion_r313582830
##
File path:
hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/TestOzoneAuditLogger.java
##
@@ -153,7 +153,7 @@ private void verifyLog(String expected) throws IOException
{
assertTrue(lines.size() != 0);
assertTrue(expected.equalsIgnoreCase(lines.get(0)));
//empty the file
-lines.remove(0);
+lines.clear();
Review comment:
@dineshchitlangia Thanks for clarifying the scope and filing the follow-up
Jira. I don't have any other concerns.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1204: HDDS-1768. Audit xxxAcl methods in OzoneManager
bharatviswa504 commented on a change in pull request #1204: HDDS-1768. Audit
xxxAcl methods in OzoneManager
URL: https://github.com/apache/hadoop/pull/1204#discussion_r313587300
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3029,24 +3044,39 @@ public boolean addAcl(OzoneObj obj, OzoneAcl acl)
throws IOException {
*/
@Override
public boolean removeAcl(OzoneObj obj, OzoneAcl acl) throws IOException {
-if(isAclEnabled) {
- checkAcls(obj.getResourceType(), obj.getStoreType(), ACLType.WRITE_ACL,
- obj.getVolumeName(), obj.getBucketName(), obj.getKeyName());
-}
-// TODO: Audit ACL operation.
-switch (obj.getResourceType()) {
-case VOLUME:
- return volumeManager.removeAcl(obj, acl);
-case BUCKET:
- return bucketManager.removeAcl(obj, acl);
-case KEY:
- return keyManager.removeAcl(obj, acl);
-case PREFIX:
- return prefixManager.removeAcl(obj, acl);
+Map auditMap = obj.toAuditMap();
+auditMap.put(OzoneConsts.ACLS, acl.toString());
Review comment:
Discussed offline, how we can make this audit for acls in to a common
method, which can help HA code path.
And also can you file a Jira to make code changes to address this in HA code
path also.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] avijayanhwx commented on issue #1268: HDDS-1910. Cannot build hadoop-hdds-config from scratch in IDEA
avijayanhwx commented on issue #1268: HDDS-1910. Cannot build hadoop-hdds-config from scratch in IDEA URL: https://github.com/apache/hadoop/pull/1268#issuecomment-520991112 LGTM +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on issue #1267: HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields
jojochuang commented on issue #1267: HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields URL: https://github.com/apache/hadoop/pull/1267#issuecomment-520991334 Ok. Let's spend some time on the doc, and raise a new jira to update the doc. +1 for the change in the PR, pending Jenkins. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang opened a new pull request #1289: HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields
jojochuang opened a new pull request #1289: HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields URL: https://github.com/apache/hadoop/pull/1289 Fork from PR #1267 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[
https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906601#comment-16906601
]
Hadoop QA commented on HADOOP-16459:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
6s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
21s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
18s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
46s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
39s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
5s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
28s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 14m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
48s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 48s{color} | {color:orange} root: The patch generated 6 new + 351 unchanged
- 7 fixed = 357 total (was 358) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
21s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
31s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 42s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}207m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoo
[jira] [Commented] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906608#comment-16906608 ] Íñigo Goiri commented on HADOOP-16459: -- The differences between [^HADOOP-16266-branch-2.002.patch] and [^HADOOP-16266-branch-3.0.000.patch] are the expected ones. The unit test are usual suspects and the checkstyle already discussed. +1 on [^HADOOP-16266-branch-2.002.patch]. > Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the > RPC layer" to branch-2 > > > Key: HADOOP-16459 > URL: https://issues.apache.org/jira/browse/HADOOP-16459 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Fix For: 3.0.4, 3.2.1, 3.1.3 > > Attachments: HADOOP-16266-branch-2.000.patch, > HADOOP-16266-branch-2.001.patch, HADOOP-16266-branch-2.002.patch, > HADOOP-16266-branch-3.0.000.patch, HADOOP-16266-branch-3.1.000.patch, > HADOOP-16266-branch-3.2.000.patch > > > We would like to target pulling HADOOP-16266, an important operability > enhancement and prerequisite for HDFS-14403, into branch-2. > It's only present in trunk now so we also need to backport through the 3.x > lines. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] LeonGao91 commented on issue #1252: HDFS-14678. Allow triggerBlockReport to a specific namenode.
LeonGao91 commented on issue #1252: HDFS-14678. Allow triggerBlockReport to a specific namenode. URL: https://github.com/apache/hadoop/pull/1252#issuecomment-521013444 Thanks @jojochuang ! looks like the # of test failures reduced to 1 with re-run, and it is not related, could you take a look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1289: HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields
hadoop-yetus commented on issue #1289: HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields URL: https://github.com/apache/hadoop/pull/1289#issuecomment-521013567 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 73 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1165 | trunk passed | | +1 | compile | 24 | trunk passed | | +1 | checkstyle | 22 | trunk passed | | +1 | mvnsite | 35 | trunk passed | | +1 | shadedclient | 840 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 21 | trunk passed | | 0 | spotbugs | 42 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 40 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 23 | the patch passed | | +1 | compile | 19 | the patch passed | | +1 | javac | 19 | the patch passed | | +1 | checkstyle | 17 | the patch passed | | +1 | mvnsite | 26 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 833 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 20 | the patch passed | | +1 | findbugs | 48 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 310 | hadoop-hdfs-httpfs in the patch passed. | | +1 | asflicense | 36 | The patch does not generate ASF License warnings. | | | | 3670 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1289/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1289 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 806f5f6be0f4 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 2ac029b | | Default Java | 1.8.0_222 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1289/1/testReport/ | | Max. process+thread count | 591 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project/hadoop-hdfs-httpfs | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1289/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on issue #1252: HDFS-14678. Allow triggerBlockReport to a specific namenode.
jojochuang commented on issue #1252: HDFS-14678. Allow triggerBlockReport to a specific namenode. URL: https://github.com/apache/hadoop/pull/1252#issuecomment-521015474 I gave it a quick review and from that quick look it looks good. Thanks. I'll try to look at it harder, especially the test code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906628#comment-16906628 ] Erik Krogen commented on HADOOP-16459: -- Thanks [~elgoiri]! Just committed this to branch-2. > Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the > RPC layer" to branch-2 > > > Key: HADOOP-16459 > URL: https://issues.apache.org/jira/browse/HADOOP-16459 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Fix For: 3.0.4, 3.2.1, 3.1.3 > > Attachments: HADOOP-16266-branch-2.000.patch, > HADOOP-16266-branch-2.001.patch, HADOOP-16266-branch-2.002.patch, > HADOOP-16266-branch-3.0.000.patch, HADOOP-16266-branch-3.1.000.patch, > HADOOP-16266-branch-3.2.000.patch > > > We would like to target pulling HADOOP-16266, an important operability > enhancement and prerequisite for HDFS-14403, into branch-2. > It's only present in trunk now so we also need to backport through the 3.x > lines. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HADOOP-16459: - Resolution: Fixed Fix Version/s: 2.10.0 Status: Resolved (was: Patch Available) > Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the > RPC layer" to branch-2 > > > Key: HADOOP-16459 > URL: https://issues.apache.org/jira/browse/HADOOP-16459 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.2.1, 3.1.3 > > Attachments: HADOOP-16266-branch-2.000.patch, > HADOOP-16266-branch-2.001.patch, HADOOP-16266-branch-2.002.patch, > HADOOP-16266-branch-3.0.000.patch, HADOOP-16266-branch-3.1.000.patch, > HADOOP-16266-branch-3.2.000.patch > > > We would like to target pulling HADOOP-16266, an important operability > enhancement and prerequisite for HDFS-14403, into branch-2. > It's only present in trunk now so we also need to backport through the 3.x > lines. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16266) Add more fine-grained processing time metrics to the RPC layer
[ https://issues.apache.org/jira/browse/HADOOP-16266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HADOOP-16266: - Fix Version/s: 2.10.0 > Add more fine-grained processing time metrics to the RPC layer > -- > > Key: HADOOP-16266 > URL: https://issues.apache.org/jira/browse/HADOOP-16266 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Christopher Gregorian >Assignee: Erik Krogen >Priority: Minor > Labels: rpc > Fix For: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 3.1.3 > > Attachments: HADOOP-16266.001.patch, HADOOP-16266.002.patch, > HADOOP-16266.003.patch, HADOOP-16266.004.patch, HADOOP-16266.005.patch, > HADOOP-16266.006.patch, HADOOP-16266.007.patch, HADOOP-16266.008.patch, > HADOOP-16266.009.patch, HADOOP-16266.010.patch, > HADOOP-16266.011-followon.patch, HADOOP-16266.011.patch > > > Splitting off of HDFS-14403 to track the first part: introduces more > fine-grained measuring of how a call's processing time is split up. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer commented on a change in pull request #1205: HDDS-1886. Use ArrayList#clear to address audit failure scenario
anuengineer commented on a change in pull request #1205: HDDS-1886. Use
ArrayList#clear to address audit failure scenario
URL: https://github.com/apache/hadoop/pull/1205#discussion_r313627412
##
File path:
hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/TestOzoneAuditLogger.java
##
@@ -153,7 +153,7 @@ private void verifyLog(String expected) throws IOException
{
assertTrue(lines.size() != 0);
assertTrue(expected.equalsIgnoreCase(lines.get(0)));
//empty the file
-lines.remove(0);
+lines.clear();
Review comment:
@adoroszlai Thanks for the review. I will commit this now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anuengineer closed pull request #1205: HDDS-1886. Use ArrayList#clear to address audit failure scenario
anuengineer closed pull request #1205: HDDS-1886. Use ArrayList#clear to address audit failure scenario URL: https://github.com/apache/hadoop/pull/1205 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] dineshchitlangia commented on issue #1204: HDDS-1768. Audit xxxAcl methods in OzoneManager
dineshchitlangia commented on issue #1204: HDDS-1768. Audit xxxAcl methods in OzoneManager URL: https://github.com/apache/hadoop/pull/1204#issuecomment-521029139 @bharatviswa504 Thanks for reviewing. Updated PR to address review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1253: HDFS-8631. WebHDFS : Support setQuota
hadoop-yetus commented on issue #1253: HDFS-8631. WebHDFS : Support setQuota URL: https://github.com/apache/hadoop/pull/1253#issuecomment-521029852 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 48 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 60 | Maven dependency ordering for branch | | +1 | mvninstall | 1102 | trunk passed | | +1 | compile | 1118 | trunk passed | | +1 | checkstyle | 159 | trunk passed | | +1 | mvnsite | 314 | trunk passed | | +1 | shadedclient | 1199 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 210 | trunk passed | | 0 | spotbugs | 79 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 539 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 23 | Maven dependency ordering for patch | | +1 | mvninstall | 202 | the patch passed | | +1 | compile | 1046 | the patch passed | | +1 | javac | 1046 | the patch passed | | -0 | checkstyle | 137 | root: The patch generated 11 new + 604 unchanged - 0 fixed = 615 total (was 604) | | +1 | mvnsite | 263 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 635 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 226 | the patch passed | | +1 | findbugs | 577 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 521 | hadoop-common in the patch failed. | | +1 | unit | 121 | hadoop-hdfs-client in the patch passed. | | -1 | unit | 6420 | hadoop-hdfs in the patch failed. | | +1 | unit | 332 | hadoop-hdfs-httpfs in the patch passed. | | -1 | unit | 1442 | hadoop-hdfs-rbf in the patch failed. | | +1 | asflicense | 66 | The patch does not generate ASF License warnings. | | | | 16588 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.util.TestReadWriteDiskValidator | | | hadoop.fs.TestFilterFileSystem | | | hadoop.fs.TestHarFileSystem | | | hadoop.hdfs.TestDFSStorageStateRecovery | | | hadoop.hdfs.TestErasureCodingPolicies | | | hadoop.hdfs.TestDFSStartupVersions | | | hadoop.fs.viewfs.TestViewFileSystemHdfs | | | hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy | | | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy | | | hadoop.hdfs.TestMaintenanceState | | | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy | | | hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy | | | hadoop.fs.TestSymlinkHdfsFileSystem | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | | hadoop.hdfs.server.namenode.snapshot.TestRandomOpsWithSnapshots | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.server.federation.router.TestRouterWithSecureStartup | | | hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1253/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1253 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux d1f25fc6ac93 4.4.0-157-generic #185-Ubuntu SMP Tue Jul 23 09:17:01 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 2ac029b | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1253/7/artifact/out/diff-checkstyle-root.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1253/7/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1253/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1253/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1253/7/testReport/ | | Max. process+thread count | 3302 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/had
