[jira] [Work logged] (MAPREDUCE-7270) TestHistoryViewerPrinter could be failed when the locale isn't English.
[ https://issues.apache.org/jira/browse/MAPREDUCE-7270?focusedWorklogId=562869=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-562869 ] ASF GitHub Bot logged work on MAPREDUCE-7270: - Author: ASF GitHub Bot Created on: 09/Mar/21 07:41 Start Date: 09/Mar/21 07:41 Worklog Time Spent: 10m Work Description: liuml07 commented on a change in pull request #1942: URL: https://github.com/apache/hadoop/pull/1942#discussion_r590030931 ## File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestHistoryViewerPrinter.java ## @@ -43,6 +46,19 @@ private final String LINE_SEPARATOR = System.lineSeparator(); + private static Locale DEFAULT_LOCALE; Review comment: This is only initialized during the static `@BeforeClass setUp()` method, and it will not change during every test cases in this class. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 562869) Time Spent: 20m (was: 10m) > TestHistoryViewerPrinter could be failed when the locale isn't English. > --- > > Key: MAPREDUCE-7270 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7270 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Sungpeo Kook >Priority: Minor > Labels: pull-request-available > Attachments: MAPREDUCE-7270.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Both of testHumanPrinter and testHumanPrinterAll have expected string for > assertion. > But the actual result includes the Dateformat which can be different depends > on Locale. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7328) Job.monitorAndPrintJob function can sleep most for 596 hours when jobclient.progress.monitor.poll.interval is misconfigured , causing the job to hang
Tina Shan created MAPREDUCE-7328:
Summary: Job.monitorAndPrintJob function can sleep most for 596
hours when jobclient.progress.monitor.poll.interval is misconfigured , causing
the job to hang
Key: MAPREDUCE-7328
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7328
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.3.0
Reporter: Tina Shan
The loop terminates depending on a configurable value and there is little
sanity checking on this value. When jobclient.progress.monitor.poll.interval is
misconfigured to INT_MAX, it can cause the loop to sleep at most for 596 hours.
The thread would get stuck and never report progress to the user even if the
job moves forward. We suggest adding a cap value or a warning message.
{code:java}
public boolean monitorAndPrintJob()
throws IOException, InterruptedException {
...
while (!isComplete() || !reportedAfterCompletion) {
if (isComplete()) {
reportedAfterCompletion = true;
} else {
Thread.sleep(progMonitorPollIntervalMillis);
}
...
}
{code}
Similar bug to MAPREDUCE-7327
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7327) Job.waitForCompletion function can sleep most for 596 hours when jobclient.completion.poll.interval is misconfigured , causing the job to hang
Tina Shan created MAPREDUCE-7327:
Summary: Job.waitForCompletion function can sleep most for 596
hours when jobclient.completion.poll.interval is misconfigured , causing the
job to hang
Key: MAPREDUCE-7327
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7327
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.3.0
Reporter: Tina Shan
The loop terminates depending on a configurable value and there is little
sanity checking on this value. When jobclient.completion.poll.interval is
misconfigured to INT_MAX, it can cause the loop to sleep at most for 596 hours.
The thread would get stuck and never return to the user even if the job
completes. We suggest adding a cap value or a warning message.
{code:java}
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
...
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
...
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297703#comment-17297703
]
Hadoop QA commented on MAPREDUCE-7322:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
47s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include
10 new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
31s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
4s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
2s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 48s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
5s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
59s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
25s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
3s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
3s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
4s{color} | {color:green}{color} | {color:green} root: The patch generated 0
new + 1163 unchanged - 18 fixed = 1163 total (was 1181) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
4s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 1s{color} | {color:green}{color} | {color:green}
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297487#comment-17297487
]
Ahmed Hussein commented on MAPREDUCE-7322:
--
List of changes:
* Moved {{TestMRIntermediateDataEncryption}} to
{{org.apache.hadoop.mapreduce}}.
* Change the implementation of {{TestMRIntermediateDataEncryption}} generating
textInput, then counting the words. The configuration are set in a way to
guarantee that Spilling is triggered.
* I had to disable multiple reducers case (a.k.a. {{"testMultipleReducers", 2,
4, false}} ) because of the bug reported in MAPREDUCE-7325.
* I made some changes in {{RandomTextWriter}} to use {{ThreadLocalRandom}} and
to expose the random sentence generator.
* Created {{SpillCallBackInjector}} injecting callbacks while spilling files.
* Instead of changing {{CryptoUtils}} - which can be tricky for backward
compatibility and ecosystem- I preferred adding a new wrapper
{{IntermediateEncryptedStream}}.
* capturing the Spilled files implied changing the calls in: {{MapTask}},
{{Merger}}, {{Fetcher}}, {{LocalFetcher}}, {{OnDiskMapOutput}},
{{BackupStore}}, {{MergeManagerImpl}}
* Added some utility methods in {{GenericTestUtils}} to organize output folders.
* Applied initializations to other unit tests related to intermediate data.
* Fixed a bug in the {{JarFinder}} where temporary file was not deleted in
{{test-dir}}
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297490#comment-17297490
]
Ahmed Hussein commented on MAPREDUCE-7322:
--
I submitted
[MAPREDUCE-7322.005.patch|https://issues.apache.org/jira/secure/attachment/13021872/MAPREDUCE-7322.005.patch]
fixing the check styles warnings.
[~Jim_Brennan] Can you please take a look at the patch?
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Attachment: MAPREDUCE-7322.005.patch
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
