[jira] [Commented] (HDFS-8178) QJM doesn't purge empty and corrupt inprogress edits files
[
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520617#comment-14520617
]
Aaron T. Myers commented on HDFS-8178:
--
Hi Zhe, thanks a lot for posting this patch. It looks pretty good to me. I have
one comment, and one question for you:
Comment: I'm a tad concerned that the current patch be be over aggressive in
terms of purging edit log segments, if somehow we ended up with two in-progress
segments, both of which were required to form the complete history of edit log
transactions. Unless we're somehow guaranteed that that's not a state we could
end up in, I think we should do something to guarantee that the in-progress
edits file(s) that we're considering purging definitely overlap with finalized
edit log segments so that they don't contain any edits that we can't afford to
lose. Or, perhaps we should move them aside with a different name until such
time as we can be sure that we don't need their transactions anymore, i.e.
their transaction is less than minTxIdToKeep, and so we can definitely safely
discard them.
Question: In the patch for HDFS-5919, we introduced a new regex for identifying
stale in-progress files. I'm not familiar with why that was necessary in that
patch, but can you please comment on why it's not necessary in this case?
Naively, I'd expect either both or neither the FJM and JN to require accounting
for that.
> QJM doesn't purge empty and corrupt inprogress edits files
> --
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: qjm
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8178.000.patch
>
>
> HDFS-5919 fixes the issue for {{FileJournalManager}}. A similar fix is needed
> for QJM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8178) QJM doesn't purge empty and corrupt inprogress edits files
[
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512079#comment-14512079
]
Zhe Zhang commented on HDFS-8178:
-
{{TestEncryptionZonesWithKMS}} is unrelated and passes locally.
> QJM doesn't purge empty and corrupt inprogress edits files
> --
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: qjm
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8178.000.patch
>
>
> HDFS-5919 fixes the issue for {{FileJournalManager}}. A similar fix is needed
> for QJM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8178) QJM doesn't purge empty and corrupt inprogress edits files
[
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508347#comment-14508347
]
Hadoop QA commented on HDFS-8178:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 38s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | javac | 7m 25s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 7m 15s | The applied patch generated 1
additional checkstyle issues. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 5s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 3m 12s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 164m 31s | Tests failed in hadoop-hdfs. |
| | | 212m 16s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestEncryptionZonesWithKMS |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12727464/HDFS-8178.000.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0ebe84d |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/10350/artifact/patchprocess/checkstyle-result-diff.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10350/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/10350/testReport/ |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10350//console |
This message was automatically generated.
> QJM doesn't purge empty and corrupt inprogress edits files
> --
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: qjm
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8178.000.patch
>
>
> HDFS-5919 fixes the issue for {{FileJournalManager}}. A similar fix is needed
> for QJM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
