subject:"\[jira\] \[Work logged\] \(HDDS\-1956\) Aged IO Thread exits on first read"

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=294463=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294463
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 14/Aug/19 05:33
Start Date: 14/Aug/19 05:33
Worklog Time Spent: 10m 
  Work Description: adoroszlai commented on issue #1287: HDDS-1956. Aged IO 
Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#issuecomment-521107553
 
 
   Thanks @mukul1987 for the review, and @anuengineer for commiting it.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 294463)
Time Spent: 2h 40m  (was: 2.5h)

> Aged IO Thread exits on first read
> --
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.5.0
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to 
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(194)) - AGED LOADGEN: Started 
> Aged IO Thread:2139.
> ...
> 2019-08-12 22:55:47,147 [pool-245-thread-1] ERROR 
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(213)) - AGED LOADGEN: 0 
> Exiting due to exception
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.readData(MiniOzoneLoadGenerator.java:151)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.startAgedFilesLoad(MiniOzoneLoadGenerator.java:209)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$1(MiniOzoneLoadGenerator.java:235)
> 2019-08-12 22:55:47,149 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(219)) - Terminating IO 
> thread:2139.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=294438=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294438
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 14/Aug/19 04:58
Start Date: 14/Aug/19 04:58
Worklog Time Spent: 10m 
  Work Description: anuengineer commented on pull request #1287: HDDS-1956. 
Aged IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 294438)
Time Spent: 2.5h  (was: 2h 20m)

> Aged IO Thread exits on first read
> --
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.5.0
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to 
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(194)) - AGED LOADGEN: Started 
> Aged IO Thread:2139.
> ...
> 2019-08-12 22:55:47,147 [pool-245-thread-1] ERROR 
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(213)) - AGED LOADGEN: 0 
> Exiting due to exception
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.readData(MiniOzoneLoadGenerator.java:151)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.startAgedFilesLoad(MiniOzoneLoadGenerator.java:209)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$1(MiniOzoneLoadGenerator.java:235)
> 2019-08-12 22:55:47,149 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(219)) - Terminating IO 
> thread:2139.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=294254=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294254
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 13/Aug/19 22:39
Start Date: 13/Aug/19 22:39
Worklog Time Spent: 10m 
  Work Description: avijayanhwx commented on issue #1287: HDDS-1956. Aged 
IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#issuecomment-521035462
 
 
   /label ozone
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 294254)
Time Spent: 2h 20m  (was: 2h 10m)

> Aged IO Thread exits on first read
> --
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.5.0
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to 
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(194)) - AGED LOADGEN: Started 
> Aged IO Thread:2139.
> ...
> 2019-08-12 22:55:47,147 [pool-245-thread-1] ERROR 
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(213)) - AGED LOADGEN: 0 
> Exiting due to exception
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.readData(MiniOzoneLoadGenerator.java:151)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.startAgedFilesLoad(MiniOzoneLoadGenerator.java:209)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$1(MiniOzoneLoadGenerator.java:235)
> 2019-08-12 22:55:47,149 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(219)) - Terminating IO 
> thread:2139.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=294085=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294085
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 13/Aug/19 18:30
Start Date: 13/Aug/19 18:30
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on issue #1287: HDDS-1956. Aged 
IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#issuecomment-520953642
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | 0 | reexec | 41 | Docker mode activated. |
   ||| _ Prechecks _ |
   | +1 | dupname | 0 | No case conflicting files found. |
   | +1 | @author | 0 | The patch does not contain any @author tags. |
   | +1 | test4tests | 0 | The patch appears to include 3 new or modified test 
files. |
   ||| _ trunk Compile Tests _ |
   | +1 | mvninstall | 639 | trunk passed |
   | +1 | compile | 389 | trunk passed |
   | +1 | checkstyle | 66 | trunk passed |
   | +1 | mvnsite | 0 | trunk passed |
   | -1 | shadedclient | 864 | branch has errors when building and testing our 
client artifacts. |
   | +1 | javadoc | 166 | trunk passed |
   | 0 | spotbugs | 465 | Used deprecated FindBugs config; considering 
switching to SpotBugs. |
   | +1 | findbugs | 670 | trunk passed |
   ||| _ Patch Compile Tests _ |
   | +1 | mvninstall | 567 | the patch passed |
   | +1 | compile | 374 | the patch passed |
   | +1 | javac | 374 | the patch passed |
   | +1 | checkstyle | 73 | the patch passed |
   | +1 | mvnsite | 0 | the patch passed |
   | +1 | whitespace | 0 | The patch has no whitespace issues. |
   | +1 | shadedclient | 691 | patch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 153 | the patch passed |
   | -1 | findbugs | 320 | hadoop-ozone in the patch failed. |
   ||| _ Other Tests _ |
   | -1 | unit | 13 | hadoop-hdds in the patch failed. |
   | -1 | unit | 31 | hadoop-ozone in the patch failed. |
   | 0 | asflicense | 31 | ASF License check generated no output? |
   | | | 5461 | |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.1 Server=19.03.1 base: 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/2/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/1287 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient findbugs checkstyle |
   | uname | Linux 512599488faa 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | personality/hadoop.sh |
   | git revision | trunk / 2ac029b |
   | Default Java | 1.8.0_212 |
   | findbugs | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/2/artifact/out/patch-findbugs-hadoop-ozone.txt
 |
   | unit | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/2/artifact/out/patch-unit-hadoop-hdds.txt
 |
   | unit | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/2/artifact/out/patch-unit-hadoop-ozone.txt
 |
   |  Test Results | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/2/testReport/ |
   | Max. process+thread count | 447 (vs. ulimit of 5500) |
   | modules | C: hadoop-ozone/integration-test U: 
hadoop-ozone/integration-test |
   | Console output | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/2/console |
   | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
   | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
   
   
   This message was automatically generated.
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 294085)
Time Spent: 2h 10m  (was: 2h)

> Aged IO Thread exits on first read
> --
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.5.0
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to 
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
>

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=294011=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294011
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 13/Aug/19 16:37
Start Date: 13/Aug/19 16:37
Worklog Time Spent: 10m 
  Work Description: adoroszlai commented on pull request #1287: HDDS-1956. 
Aged IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313497795
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
 ##
 @@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
   String keyName = null;
   try {
 if (agedWriteProbability.isTrue()) {
-  keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+  keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
   agedLoadBucket, threadName);
 } else {
-  keyName = getKeyToRead();
-  if (keyName != null) {
+  Optional index = randomKeyToRead();
+  if (index.isPresent()) {
+keyName = getKeyName(index.get(), threadName);
 readData(agedLoadBucket, keyName);
 
 Review comment:
   Thanks for the idea, pushed a commit with this change.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 294011)
Time Spent: 2h  (was: 1h 50m)

> Aged IO Thread exits on first read
> --
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.5.0
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to 
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(194)) - AGED LOADGEN: Started 
> Aged IO Thread:2139.
> ...
> 2019-08-12 22:55:47,147 [pool-245-thread-1] ERROR 
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(213)) - AGED LOADGEN: 0 
> Exiting due to exception
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.readData(MiniOzoneLoadGenerator.java:151)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.startAgedFilesLoad(MiniOzoneLoadGenerator.java:209)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$1(MiniOzoneLoadGenerator.java:235)
> 2019-08-12 22:55:47,149 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(219)) - Terminating IO 
> thread:2139.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=294009=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294009
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 13/Aug/19 16:35
Start Date: 13/Aug/19 16:35
Worklog Time Spent: 10m 
  Work Description: mukul1987 commented on pull request #1287: HDDS-1956. 
Aged IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313496805
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
 ##
 @@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
   String keyName = null;
   try {
 if (agedWriteProbability.isTrue()) {
-  keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+  keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
 
 Review comment:
   got that, Thanks for the explanation. This comment can be ignored.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 294009)
Time Spent: 1h 50m  (was: 1h 40m)

> Aged IO Thread exits on first read
> --
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.5.0
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to 
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(194)) - AGED LOADGEN: Started 
> Aged IO Thread:2139.
> ...
> 2019-08-12 22:55:47,147 [pool-245-thread-1] ERROR 
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(213)) - AGED LOADGEN: 0 
> Exiting due to exception
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.readData(MiniOzoneLoadGenerator.java:151)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.startAgedFilesLoad(MiniOzoneLoadGenerator.java:209)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$1(MiniOzoneLoadGenerator.java:235)
> 2019-08-12 22:55:47,149 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(219)) - Terminating IO 
> thread:2139.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=294008=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294008
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 13/Aug/19 16:32
Start Date: 13/Aug/19 16:32
Worklog Time Spent: 10m 
  Work Description: adoroszlai commented on pull request #1287: HDDS-1956. 
Aged IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313494917
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
 ##
 @@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
   String keyName = null;
   try {
 if (agedWriteProbability.isTrue()) {
-  keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+  keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
 
 Review comment:
   Before key with index=0 is written, the current index is incremented to 1.  
So `randomKeyToRead` only sees `currentIndex==0` if there is no key yet.  
Please see my [example 
log](https://github.com/apache/hadoop/pull/1287#issue-306801835) for reads of 
the first key right after the first write (and even after the second one).
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 294008)
Time Spent: 1h 40m  (was: 1.5h)

> Aged IO Thread exits on first read
> --
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.5.0
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to 
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(194)) - AGED LOADGEN: Started 
> Aged IO Thread:2139.
> ...
> 2019-08-12 22:55:47,147 [pool-245-thread-1] ERROR 
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(213)) - AGED LOADGEN: 0 
> Exiting due to exception
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.readData(MiniOzoneLoadGenerator.java:151)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.startAgedFilesLoad(MiniOzoneLoadGenerator.java:209)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$1(MiniOzoneLoadGenerator.java:235)
> 2019-08-12 22:55:47,149 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(219)) - Terminating IO 
> thread:2139.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=294006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294006
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 13/Aug/19 16:31
Start Date: 13/Aug/19 16:31
Worklog Time Spent: 10m 
  Work Description: adoroszlai commented on pull request #1287: HDDS-1956. 
Aged IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313494917
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
 ##
 @@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
   String keyName = null;
   try {
 if (agedWriteProbability.isTrue()) {
-  keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+  keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
 
 Review comment:
   Before key with index=0 is written, the current index is incremented to 1.  
Please see my [example 
log](https://github.com/apache/hadoop/pull/1287#issue-306801835) for reads of 
the first key right after the first write (and even after the second one).
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 294006)
Time Spent: 1.5h  (was: 1h 20m)

> Aged IO Thread exits on first read
> --
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.5.0
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to 
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(194)) - AGED LOADGEN: Started 
> Aged IO Thread:2139.
> ...
> 2019-08-12 22:55:47,147 [pool-245-thread-1] ERROR 
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(213)) - AGED LOADGEN: 0 
> Exiting due to exception
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.readData(MiniOzoneLoadGenerator.java:151)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.startAgedFilesLoad(MiniOzoneLoadGenerator.java:209)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$1(MiniOzoneLoadGenerator.java:235)
> 2019-08-12 22:55:47,149 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(219)) - Terminating IO 
> thread:2139.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=293992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293992
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 13/Aug/19 16:15
Start Date: 13/Aug/19 16:15
Worklog Time Spent: 10m 
  Work Description: mukul1987 commented on pull request #1287: HDDS-1956. 
Aged IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313487361
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
 ##
 @@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
   String keyName = null;
   try {
 if (agedWriteProbability.isTrue()) {
-  keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+  keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
 
 Review comment:
   With getAndIncrement, the first key is written with index 0.
   in randomKeyToRead, for current index of 0, we return the Optional.empty(). 
And hence the first key will never be read.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 293992)
Time Spent: 1h 20m  (was: 1h 10m)

> Aged IO Thread exits on first read
> --
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.5.0
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to 
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(194)) - AGED LOADGEN: Started 
> Aged IO Thread:2139.
> ...
> 2019-08-12 22:55:47,147 [pool-245-thread-1] ERROR 
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(213)) - AGED LOADGEN: 0 
> Exiting due to exception
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.readData(MiniOzoneLoadGenerator.java:151)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.startAgedFilesLoad(MiniOzoneLoadGenerator.java:209)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$1(MiniOzoneLoadGenerator.java:235)
> 2019-08-12 22:55:47,149 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(219)) - Terminating IO 
> thread:2139.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=293971=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293971
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 13/Aug/19 15:40
Start Date: 13/Aug/19 15:40
Worklog Time Spent: 10m 
  Work Description: adoroszlai commented on pull request #1287: HDDS-1956. 
Aged IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313470272
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
 ##
 @@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
   String keyName = null;
   try {
 if (agedWriteProbability.isTrue()) {
-  keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+  keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
 
 Review comment:
   Can you please clarify what's the issue here?
   
   Random index selection gives a number in `[0, currentIndex)` (see 
`RandomUtils.nextInt(0, currentIndex)`).
   
   Previously files were written starting at 1 and `currentIndex` equaled the 
last written key index.  Now `currentIndex` is last written key index + 1, so 
the random choice always produces an existing key index.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 293971)
Time Spent: 1h 10m  (was: 1h)

> Aged IO Thread exits on first read
> --
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.5.0
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to 
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(194)) - AGED LOADGEN: Started 
> Aged IO Thread:2139.
> ...
> 2019-08-12 22:55:47,147 [pool-245-thread-1] ERROR 
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(213)) - AGED LOADGEN: 0 
> Exiting due to exception
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.readData(MiniOzoneLoadGenerator.java:151)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.startAgedFilesLoad(MiniOzoneLoadGenerator.java:209)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$1(MiniOzoneLoadGenerator.java:235)
> 2019-08-12 22:55:47,149 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(219)) - Terminating IO 
> thread:2139.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=293967=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293967
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 13/Aug/19 15:35
Start Date: 13/Aug/19 15:35
Worklog Time Spent: 10m 
  Work Description: adoroszlai commented on pull request #1287: HDDS-1956. 
Aged IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313467570
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
 ##
 @@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
   String keyName = null;
   try {
 if (agedWriteProbability.isTrue()) {
-  keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+  keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
   agedLoadBucket, threadName);
 } else {
-  keyName = getKeyToRead();
-  if (keyName != null) {
+  Optional index = randomKeyToRead();
+  if (index.isPresent()) {
+keyName = getKeyName(index.get(), threadName);
 readData(agedLoadBucket, keyName);
 
 Review comment:
   OK, makes sense.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 293967)
Time Spent: 1h  (was: 50m)

> Aged IO Thread exits on first read
> --
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.5.0
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to 
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(194)) - AGED LOADGEN: Started 
> Aged IO Thread:2139.
> ...
> 2019-08-12 22:55:47,147 [pool-245-thread-1] ERROR 
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(213)) - AGED LOADGEN: 0 
> Exiting due to exception
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.readData(MiniOzoneLoadGenerator.java:151)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.startAgedFilesLoad(MiniOzoneLoadGenerator.java:209)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$1(MiniOzoneLoadGenerator.java:235)
> 2019-08-12 22:55:47,149 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(219)) - Terminating IO 
> thread:2139.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=293963=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293963
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 13/Aug/19 15:33
Start Date: 13/Aug/19 15:33
Worklog Time Spent: 10m 
  Work Description: mukul1987 commented on pull request #1287: HDDS-1956. 
Aged IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313465776
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
 ##
 @@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
   String keyName = null;
   try {
 if (agedWriteProbability.isTrue()) {
-  keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+  keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
 
 Review comment:
   Aged file index always start with 0, so we will never read the first file.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 293963)
Time Spent: 50m  (was: 40m)

> Aged IO Thread exits on first read
> --
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.5.0
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to 
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(194)) - AGED LOADGEN: Started 
> Aged IO Thread:2139.
> ...
> 2019-08-12 22:55:47,147 [pool-245-thread-1] ERROR 
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(213)) - AGED LOADGEN: 0 
> Exiting due to exception
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.readData(MiniOzoneLoadGenerator.java:151)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.startAgedFilesLoad(MiniOzoneLoadGenerator.java:209)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$1(MiniOzoneLoadGenerator.java:235)
> 2019-08-12 22:55:47,149 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(219)) - Terminating IO 
> thread:2139.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=293962=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293962
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 13/Aug/19 15:33
Start Date: 13/Aug/19 15:33
Worklog Time Spent: 10m 
  Work Description: mukul1987 commented on pull request #1287: HDDS-1956. 
Aged IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#discussion_r313464607
 
 

 ##
 File path: 
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
 ##
 @@ -201,11 +210,12 @@ private void startAgedFilesLoad(long runTimeMillis) {
   String keyName = null;
   try {
 if (agedWriteProbability.isTrue()) {
-  keyName = writeData(agedFileWrittenIndex.incrementAndGet(),
+  keyName = writeData(agedFileWrittenIndex.getAndIncrement(),
   agedLoadBucket, threadName);
 } else {
-  keyName = getKeyToRead();
-  if (keyName != null) {
+  Optional index = randomKeyToRead();
+  if (index.isPresent()) {
+keyName = getKeyName(index.get(), threadName);
 readData(agedLoadBucket, keyName);
 
 Review comment:
   lets pass the index to readData, so that we can eliminate the split function 
inside read.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 293962)
Time Spent: 40m  (was: 0.5h)

> Aged IO Thread exits on first read
> --
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.5.0
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to 
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(194)) - AGED LOADGEN: Started 
> Aged IO Thread:2139.
> ...
> 2019-08-12 22:55:47,147 [pool-245-thread-1] ERROR 
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(213)) - AGED LOADGEN: 0 
> Exiting due to exception
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.readData(MiniOzoneLoadGenerator.java:151)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.startAgedFilesLoad(MiniOzoneLoadGenerator.java:209)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$1(MiniOzoneLoadGenerator.java:235)
> 2019-08-12 22:55:47,149 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(219)) - Terminating IO 
> thread:2139.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=293807=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293807
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 13/Aug/19 10:40
Start Date: 13/Aug/19 10:40
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on issue #1287: HDDS-1956. Aged 
IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#issuecomment-520783651
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | 0 | reexec | 42 | Docker mode activated. |
   ||| _ Prechecks _ |
   | +1 | dupname | 0 | No case conflicting files found. |
   | +1 | @author | 0 | The patch does not contain any @author tags. |
   | +1 | test4tests | 0 | The patch appears to include 3 new or modified test 
files. |
   ||| _ trunk Compile Tests _ |
   | +1 | mvninstall | 614 | trunk passed |
   | +1 | compile | 368 | trunk passed |
   | +1 | checkstyle | 65 | trunk passed |
   | +1 | mvnsite | 0 | trunk passed |
   | +1 | shadedclient | 842 | branch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 167 | trunk passed |
   | 0 | spotbugs | 427 | Used deprecated FindBugs config; considering 
switching to SpotBugs. |
   | +1 | findbugs | 626 | trunk passed |
   ||| _ Patch Compile Tests _ |
   | +1 | mvninstall | 560 | the patch passed |
   | +1 | compile | 376 | the patch passed |
   | +1 | javac | 376 | the patch passed |
   | +1 | checkstyle | 80 | the patch passed |
   | +1 | mvnsite | 0 | the patch passed |
   | +1 | whitespace | 0 | The patch has no whitespace issues. |
   | +1 | shadedclient | 684 | patch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 171 | the patch passed |
   | +1 | findbugs | 651 | the patch passed |
   ||| _ Other Tests _ |
   | +1 | unit | 290 | hadoop-hdds in the patch passed. |
   | -1 | unit | 1660 | hadoop-ozone in the patch failed. |
   | +1 | asflicense | 52 | The patch does not generate ASF License warnings. |
   | | | 7432 | |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption |
   |   | hadoop.ozone.client.rpc.TestOzoneRpcClient |
   |   | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient |
   |   | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.1 Server=19.03.1 base: 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/1287 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient findbugs checkstyle |
   | uname | Linux abd4695b6acf 4.4.0-157-generic #185-Ubuntu SMP Tue Jul 23 
09:17:01 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | personality/hadoop.sh |
   | git revision | trunk / 454420e |
   | Default Java | 1.8.0_212 |
   | unit | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/1/artifact/out/patch-unit-hadoop-ozone.txt
 |
   |  Test Results | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/1/testReport/ |
   | Max. process+thread count | 5125 (vs. ulimit of 5500) |
   | modules | C: hadoop-ozone/integration-test U: 
hadoop-ozone/integration-test |
   | Console output | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1287/1/console |
   | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
   | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
   
   
   This message was automatically generated.
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 293807)
Time Spent: 0.5h  (was: 20m)

> Aged IO Thread exits on first read
> --
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.5.0
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to 
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO  
>

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=293721=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293721
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 13/Aug/19 08:35
Start Date: 13/Aug/19 08:35
Worklog Time Spent: 10m 
  Work Description: adoroszlai commented on issue #1287: HDDS-1956. Aged IO 
Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287#issuecomment-520743106
 
 
   @mukul1987 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 293721)
Time Spent: 20m  (was: 10m)

> Aged IO Thread exits on first read
> --
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.5.0
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to 
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(194)) - AGED LOADGEN: Started 
> Aged IO Thread:2139.
> ...
> 2019-08-12 22:55:47,147 [pool-245-thread-1] ERROR 
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(213)) - AGED LOADGEN: 0 
> Exiting due to exception
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.readData(MiniOzoneLoadGenerator.java:151)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.startAgedFilesLoad(MiniOzoneLoadGenerator.java:209)
>   at 
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$1(MiniOzoneLoadGenerator.java:235)
> 2019-08-12 22:55:47,149 [pool-245-thread-1] INFO  
> ozone.MiniOzoneLoadGenerator 
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(219)) - Terminating IO 
> thread:2139.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

2019-08-13 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=293720=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293720
 ]

ASF GitHub Bot logged work on HDDS-1956:


Author: ASF GitHub Bot
Created on: 13/Aug/19 08:35
Start Date: 13/Aug/19 08:35
Worklog Time Spent: 10m 
  Work Description: adoroszlai commented on pull request #1287: HDDS-1956. 
Aged IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287
 
 
   ## What changes were proposed in this pull request?
   
   1. Aged IO thread exits on first read due to 
`ArrayIndexOutOfBoundsException`.  This is caused by using `index` as the key 
name, while `readKey` expects a string separated by `_`.
   2. After fixing the above, it randomly exits due to `OMException: Key not 
found`.  This is an off-by-one error.  The first key written has `index=1`, but 
the test randomly attempts to read `index=0`.
   3. Add trace level message for read/write/delete operations.
   4. Include thread name in log output pattern for `integration-test`.
   5. Fix some log messages in `MiniOzoneChaosCluster`
   
   https://issues.apache.org/jira/browse/HDDS-1956
   
   ## How was this patch tested?
   
   Manually ran `TestMiniChaosOzoneCluster`:
   
   ```
   mvn -Phdds -pl :hadoop-ozone-integration-test 
-Dtest=TestMiniChaosOzoneCluster test
   ```
   
   Stopped it after a few minutes (see 
[HDDS-1952](https://issues.apache.org/jira/browse/HDDS-1952)).
   
   Verified that aged IO thread was writing and reading keys (until it started 
failing due to chaos):
   
   ```
   2019-08-13 10:19:18,307 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(138)) - 
LOADGEN: Writing key pool-245-thread-1_0
   2019-08-13 10:19:18,433 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(143)) - 
LOADGEN: Written key pool-245-thread-1_0
   2019-08-13 10:19:22,473 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - 
LOADGEN: Reading key pool-245-thread-1_0
   2019-08-13 10:19:22,992 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - 
LOADGEN: Read key pool-245-thread-1_0
   ...
   2019-08-13 10:19:23,837 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - 
LOADGEN: Reading key pool-245-thread-1_0
   2019-08-13 10:19:23,866 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - 
LOADGEN: Read key pool-245-thread-1_0
   2019-08-13 10:19:23,866 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(138)) - 
LOADGEN: Writing key pool-245-thread-1_1
   2019-08-13 10:19:23,870 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(143)) - 
LOADGEN: Written key pool-245-thread-1_1
   2019-08-13 10:19:23,891 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - 
LOADGEN: Reading key pool-245-thread-1_1
   2019-08-13 10:19:23,933 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - 
LOADGEN: Read key pool-245-thread-1_1
   2019-08-13 10:19:23,933 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - 
LOADGEN: Reading key pool-245-thread-1_0
   2019-08-13 10:19:23,960 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - 
LOADGEN: Read key pool-245-thread-1_0
   2019-08-13 10:19:23,960 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - 
LOADGEN: Reading key pool-245-thread-1_0
   2019-08-13 10:19:23,974 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - 
LOADGEN: Read key pool-245-thread-1_0
   2019-08-13 10:19:23,974 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - 
LOADGEN: Reading key pool-245-thread-1_0
   2019-08-13 10:19:24,017 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - 
LOADGEN: Read key pool-245-thread-1_0
   2019-08-13 10:19:24,017 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) - 
LOADGEN: Reading key pool-245-thread-1_1
   2019-08-13 10:19:24,044 [pool-245-thread-1] TRACE 
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) - 
LOADGEN: Read key pool-245-thread-1_1
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

[jira] [Work logged] (HDDS-1956) Aged IO Thread exits on first read

16 matches

Site Navigation

Mail list logo

Footer information