[
https://issues.apache.org/jira/browse/HDDS-1956?focusedWorklogId=293720&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293720
]
ASF GitHub Bot logged work on HDDS-1956:
----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Aug/19 08:35
Start Date: 13/Aug/19 08:35
Worklog Time Spent: 10m
Work Description: adoroszlai commented on pull request #1287: HDDS-1956.
Aged IO Thread exits on first read
URL: https://github.com/apache/hadoop/pull/1287
## What changes were proposed in this pull request?
1. Aged IO thread exits on first read due to
`ArrayIndexOutOfBoundsException`. This is caused by using `index` as the key
name, while `readKey` expects a string separated by `_`.
2. After fixing the above, it randomly exits due to `OMException: Key not
found`. This is an off-by-one error. The first key written has `index=1`, but
the test randomly attempts to read `index=0`.
3. Add trace level message for read/write/delete operations.
4. Include thread name in log output pattern for `integration-test`.
5. Fix some log messages in `MiniOzoneChaosCluster`
https://issues.apache.org/jira/browse/HDDS-1956
## How was this patch tested?
Manually ran `TestMiniChaosOzoneCluster`:
```
mvn -Phdds -pl :hadoop-ozone-integration-test
-Dtest=TestMiniChaosOzoneCluster test
```
Stopped it after a few minutes (see
[HDDS-1952](https://issues.apache.org/jira/browse/HDDS-1952)).
Verified that aged IO thread was writing and reading keys (until it started
failing due to chaos):
```
2019-08-13 10:19:18,307 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(138)) -
LOADGEN: Writing key pool-245-thread-1_0
2019-08-13 10:19:18,433 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(143)) -
LOADGEN: Written key pool-245-thread-1_0
2019-08-13 10:19:22,473 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) -
LOADGEN: Reading key pool-245-thread-1_0
2019-08-13 10:19:22,992 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) -
LOADGEN: Read key pool-245-thread-1_0
...
2019-08-13 10:19:23,837 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) -
LOADGEN: Reading key pool-245-thread-1_0
2019-08-13 10:19:23,866 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) -
LOADGEN: Read key pool-245-thread-1_0
2019-08-13 10:19:23,866 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(138)) -
LOADGEN: Writing key pool-245-thread-1_1
2019-08-13 10:19:23,870 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:writeData(143)) -
LOADGEN: Written key pool-245-thread-1_1
2019-08-13 10:19:23,891 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) -
LOADGEN: Reading key pool-245-thread-1_1
2019-08-13 10:19:23,933 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) -
LOADGEN: Read key pool-245-thread-1_1
2019-08-13 10:19:23,933 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) -
LOADGEN: Reading key pool-245-thread-1_0
2019-08-13 10:19:23,960 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) -
LOADGEN: Read key pool-245-thread-1_0
2019-08-13 10:19:23,960 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) -
LOADGEN: Reading key pool-245-thread-1_0
2019-08-13 10:19:23,974 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) -
LOADGEN: Read key pool-245-thread-1_0
2019-08-13 10:19:23,974 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) -
LOADGEN: Reading key pool-245-thread-1_0
2019-08-13 10:19:24,017 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) -
LOADGEN: Read key pool-245-thread-1_0
2019-08-13 10:19:24,017 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(154)) -
LOADGEN: Reading key pool-245-thread-1_1
2019-08-13 10:19:24,044 [pool-245-thread-1] TRACE
ozone.MiniOzoneLoadGenerator (MiniOzoneLoadGenerator.java:readData(176)) -
LOADGEN: Read key pool-245-thread-1_1
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 293720)
Time Spent: 10m
Remaining Estimate: 0h
> Aged IO Thread exits on first read
> ----------------------------------
>
> Key: HDDS-1956
> URL: https://issues.apache.org/jira/browse/HDDS-1956
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
> Affects Versions: 0.5.0
> Reporter: Doroszlai, Attila
> Assignee: Doroszlai, Attila
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Aged IO Thread in {{TestMiniChaosOzoneCluster}} exits on first read due to
> exception:
> {code}
> 2019-08-12 22:55:37,799 [pool-245-thread-1] INFO
> ozone.MiniOzoneLoadGenerator
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(194)) - AGED LOADGEN: Started
> Aged IO Thread:2139.
> ...
> 2019-08-12 22:55:47,147 [pool-245-thread-1] ERROR
> ozone.MiniOzoneLoadGenerator
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(213)) - AGED LOADGEN: 0
> Exiting due to exception
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.readData(MiniOzoneLoadGenerator.java:151)
> at
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.startAgedFilesLoad(MiniOzoneLoadGenerator.java:209)
> at
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$1(MiniOzoneLoadGenerator.java:235)
> 2019-08-12 22:55:47,149 [pool-245-thread-1] INFO
> ozone.MiniOzoneLoadGenerator
> (MiniOzoneLoadGenerator.java:startAgedFilesLoad(219)) - Terminating IO
> thread:2139.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
