[jira] [Commented] (HADOOP-14508) TestDFSIO throws NPE when set -sequential argument.
[
https://issues.apache.org/jira/browse/HADOOP-14508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899741#comment-16899741
]
Wei-Chiu Chuang commented on HADOOP-14508:
--
i reviewed this patch but to be honest i don't know what -seq is for.
Somebody should write a better doc or help text about this tool.
> TestDFSIO throws NPE when set -sequential argument.
> ---
>
> Key: HADOOP-14508
> URL: https://issues.apache.org/jira/browse/HADOOP-14508
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha4
>Reporter: Wenxin He
>Assignee: Wenxin He
>Priority: Major
> Attachments: HADOOP-14508.002.patch
>
>
> Benchmark tool TestDFSIO throws NPE when set {{-sequential}} due to
> uninitialized {{ioer.stream}} in {{TestDFSIO#sequentialTest}}.
> More descriptions, stack traces see comments.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16487) Update jackson-databind to 2.9.9.2
[ https://issues.apache.org/jira/browse/HADOOP-16487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899739#comment-16899739 ] Hudson commented on HADOOP-16487: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17038 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17038/]) HADOOP-16487. Update jackson-databind to 2.9.9.2. Contributed by Siyao (weichiu: rev 9680a8b2371c2ff8b5314e8fa39bdb9f8db46f96) * (edit) hadoop-project/pom.xml > Update jackson-databind to 2.9.9.2 > -- > > Key: HADOOP-16487 > URL: https://issues.apache.org/jira/browse/HADOOP-16487 > Project: Hadoop Common > Issue Type: Bug >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Critical > Fix For: 3.3.0 > > Attachments: HADOOP-16487.001.patch > > > Another CVE in jackson-databind: > https://nvd.nist.gov/vuln/detail/CVE-2019-14379 > jackson-databind 2.9.9.2 is available: > https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind > Side note: Here's a discussion jira on whether to remove jackson-databind due > to the increasing number of CVEs in this dependency recently: HADOOP-16485 -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16487) Update jackson-databind to 2.9.9.2
[ https://issues.apache.org/jira/browse/HADOOP-16487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-16487: - Resolution: Fixed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) Pushed to trunk. Now we should get HADOOP-16451 and this one both into lower branches too. > Update jackson-databind to 2.9.9.2 > -- > > Key: HADOOP-16487 > URL: https://issues.apache.org/jira/browse/HADOOP-16487 > Project: Hadoop Common > Issue Type: Bug >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Critical > Fix For: 3.3.0 > > Attachments: HADOOP-16487.001.patch > > > Another CVE in jackson-databind: > https://nvd.nist.gov/vuln/detail/CVE-2019-14379 > jackson-databind 2.9.9.2 is available: > https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind > Side note: Here's a discussion jira on whether to remove jackson-databind due > to the increasing number of CVEs in this dependency recently: HADOOP-16485 -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16336) finish variable is unused in ZStandardCompressor
[
https://issues.apache.org/jira/browse/HADOOP-16336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899731#comment-16899731
]
xuzq commented on HADOOP-16336:
---
Thanks for the revert, [~jojochuang].
If there are multiple Frames in ZSTD's data, Decompression operation will be
failed.
I created one Jira to track this bug.
[HDFS-14099|https://issues.apache.org/jira/browse/HDFS-14099].
> finish variable is unused in ZStandardCompressor
>
>
> Key: HADOOP-16336
> URL: https://issues.apache.org/jira/browse/HADOOP-16336
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Daniel Templeton
>Priority: Trivial
> Labels: newbie
> Fix For: 3.3.0
>
>
> The boolean {{finish}} variable is unused and can be removed:
> {code:java}
> private boolean finish, finished;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Apache9 commented on issue #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c…
Apache9 commented on issue #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c… URL: https://github.com/apache/hadoop/pull/1222#issuecomment-518055882 > For all the changes, I think we should add tests verifying consistent loading of existing classes first. That is: we save the current .json files as test resources and verify that they can be loaded with the new code. That's a good practise anyway, isn't it? This seems to be a good idea. Will have a try after I finish purging the jackson stuff. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Apache9 commented on a change in pull request #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c…
Apache9 commented on a change in pull request #1222: HADOOP-16488 Deprecated
JsonSerialization and move it out of hadoop-c…
URL: https://github.com/apache/hadoop/pull/1222#discussion_r310416038
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
##
@@ -3718,52 +3709,42 @@ public static void dumpConfiguration(Configuration
config,
*/
public static void dumpConfiguration(Configuration config,
Writer out) throws IOException {
-JsonFactory dumpFactory = new JsonFactory();
-JsonGenerator dumpGenerator = dumpFactory.createGenerator(out);
-dumpGenerator.writeStartObject();
-dumpGenerator.writeFieldName("properties");
-dumpGenerator.writeStartArray();
-dumpGenerator.flush();
+JsonWriter jsonWriter = new JsonWriter(out);
+jsonWriter.beginObject().name("properties").beginArray().flush();
Review comment:
We have, but as you said below, if we purge jackson completely, there is no
way to ensure that the json format is not changed, as in tests we also use
gson...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Apache9 commented on a change in pull request #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c…
Apache9 commented on a change in pull request #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c… URL: https://github.com/apache/hadoop/pull/1222#discussion_r310415981 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java ## @@ -106,12 +106,6 @@ import org.w3c.dom.Document; import org.w3c.dom.Element; -import com.google.common.base.Preconditions; -import com.google.common.base.Strings; - -import static org.apache.commons.lang3.StringUtils.isBlank; -import static org.apache.commons.lang3.StringUtils.isNotBlank; - Review comment: Is there an import order rule? I checked several files but can not find the pattern, static import can be anywhere, and no alphabetical order... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on issue #1225: HDDS-1909. Use new HA code for Non-HA in OM.
bharatviswa504 commented on issue #1225: HDDS-1909. Use new HA code for Non-HA in OM. URL: https://github.com/apache/hadoop/pull/1225#issuecomment-518050678 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1225: HDDS-1909. Use new HA code for Non-HA in OM.
hadoop-yetus commented on issue #1225: HDDS-1909. Use new HA code for Non-HA in OM. URL: https://github.com/apache/hadoop/pull/1225#issuecomment-518049935 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 81 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 745 | trunk passed | | +1 | compile | 418 | trunk passed | | +1 | checkstyle | 77 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 851 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 184 | trunk passed | | 0 | spotbugs | 521 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 756 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 664 | the patch passed | | +1 | compile | 445 | the patch passed | | +1 | javac | 445 | the patch passed | | -0 | checkstyle | 45 | hadoop-ozone: The patch generated 22 new + 0 unchanged - 0 fixed = 22 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 771 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 184 | the patch passed | | +1 | findbugs | 797 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 376 | hadoop-hdds in the patch passed. | | -1 | unit | 320 | hadoop-ozone in the patch failed. | | +1 | asflicense | 36 | The patch does not generate ASF License warnings. | | | | 6929 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.request.s3.bucket.TestS3BucketCreateRequest | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1225 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux bca472d92b43 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 065cbc6 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/4/artifact/out/diff-checkstyle-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/4/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/4/testReport/ | | Max. process+thread count | 1392 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1225: HDDS-1909. Use new HA code for Non-HA in OM.
hadoop-yetus commented on issue #1225: HDDS-1909. Use new HA code for Non-HA in OM. URL: https://github.com/apache/hadoop/pull/1225#issuecomment-518049554 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 83 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 697 | trunk passed | | +1 | compile | 412 | trunk passed | | +1 | checkstyle | 80 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 967 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 177 | trunk passed | | 0 | spotbugs | 519 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 799 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 631 | the patch passed | | +1 | compile | 412 | the patch passed | | +1 | javac | 412 | the patch passed | | -0 | checkstyle | 44 | hadoop-ozone: The patch generated 22 new + 0 unchanged - 0 fixed = 22 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 660 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 205 | the patch passed | | +1 | findbugs | 774 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 249 | hadoop-hdds in the patch failed. | | -1 | unit | 301 | hadoop-ozone in the patch failed. | | +1 | asflicense | 41 | The patch does not generate ASF License warnings. | | | | 6717 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.om.request.s3.bucket.TestS3BucketCreateRequest | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1225 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux de11ea326691 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 065cbc6 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/3/artifact/out/diff-checkstyle-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/3/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/3/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/3/testReport/ | | Max. process+thread count | 1384 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1225: HDDS-1909. Use new HA code for Non-HA in OM.
hadoop-yetus commented on issue #1225: HDDS-1909. Use new HA code for Non-HA in OM. URL: https://github.com/apache/hadoop/pull/1225#issuecomment-518048715 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 44 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 599 | trunk passed | | +1 | compile | 366 | trunk passed | | +1 | checkstyle | 70 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 847 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 164 | trunk passed | | 0 | spotbugs | 420 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 612 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 543 | the patch passed | | +1 | compile | 372 | the patch passed | | +1 | javac | 372 | the patch passed | | -0 | checkstyle | 42 | hadoop-ozone: The patch generated 22 new + 0 unchanged - 0 fixed = 22 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 683 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 198 | the patch passed | | +1 | findbugs | 631 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 291 | hadoop-hdds in the patch passed. | | -1 | unit | 168 | hadoop-ozone in the patch failed. | | +1 | asflicense | 43 | The patch does not generate ASF License warnings. | | | | 5879 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.request.s3.bucket.TestS3BucketCreateRequest | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1225 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 034edb50e155 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 065cbc6 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/5/artifact/out/diff-checkstyle-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/5/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/5/testReport/ | | Max. process+thread count | 1321 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/5/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1225: HDDS-1909. Use new HA code for Non-HA in OM.
hadoop-yetus commented on issue #1225: HDDS-1909. Use new HA code for Non-HA in OM. URL: https://github.com/apache/hadoop/pull/1225#issuecomment-518048288 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 42 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 596 | trunk passed | | +1 | compile | 369 | trunk passed | | +1 | checkstyle | 75 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 878 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 164 | trunk passed | | 0 | spotbugs | 421 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 614 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 549 | the patch passed | | +1 | compile | 380 | the patch passed | | +1 | javac | 380 | the patch passed | | -0 | checkstyle | 44 | hadoop-ozone: The patch generated 22 new + 0 unchanged - 0 fixed = 22 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 1 | The patch has no whitespace issues. | | +1 | shadedclient | 678 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 166 | the patch passed | | +1 | findbugs | 649 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 304 | hadoop-hdds in the patch failed. | | -1 | unit | 165 | hadoop-ozone in the patch failed. | | +1 | asflicense | 43 | The patch does not generate ASF License warnings. | | | | 5922 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.block.TestBlockManager | | | hadoop.ozone.om.request.s3.bucket.TestS3BucketCreateRequest | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1225 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux a494df207080 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 065cbc6 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/2/artifact/out/diff-checkstyle-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/2/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/2/testReport/ | | Max. process+thread count | 1412 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on issue #1225: HDDS-1909. Use new HA code for Non-HA in OM.
bharatviswa504 commented on issue #1225: HDDS-1909. Use new HA code for Non-HA in OM. URL: https://github.com/apache/hadoop/pull/1225#issuecomment-518042798 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1225: HDDS-1909. Use new HA code for Non-HA in OM.
hadoop-yetus commented on issue #1225: HDDS-1909. Use new HA code for Non-HA in OM. URL: https://github.com/apache/hadoop/pull/1225#issuecomment-518041536 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 13 | https://github.com/apache/hadoop/pull/1225 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/1225 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/1/console | | versions | git=2.7.4 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 opened a new pull request #1225: HDDS-1909. Use new HA code for Non-HA in OM.
bharatviswa504 opened a new pull request #1225: HDDS-1909. Use new HA code for Non-HA in OM. URL: https://github.com/apache/hadoop/pull/1225 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1221: HDDS-1488. Scm cli command to start/stop replication manager.
bharatviswa504 commented on a change in pull request #1221: HDDS-1488. Scm cli
command to start/stop replication manager.
URL: https://github.com/apache/hadoop/pull/1221#discussion_r310406231
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ReplicationManager.java
##
@@ -154,6 +151,9 @@ public synchronized void start() {
if (!running) {
Review comment:
Here we need to check isRunning() method.
Take an example:
call 1. start, 2. stop, (but it has not stopped completely) 3. calling start
immediately we try to start again if we only check running flag.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1221: HDDS-1488. Scm cli command to start/stop replication manager.
bharatviswa504 commented on a change in pull request #1221: HDDS-1488. Scm cli
command to start/stop replication manager.
URL: https://github.com/apache/hadoop/pull/1221#discussion_r310406231
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ReplicationManager.java
##
@@ -154,6 +151,9 @@ public synchronized void start() {
if (!running) {
Review comment:
Here we need to check isRunning() method.
Take an example:
call start, stop, (but it has not stopped completely, calling start
immediately we try to start again if we only check running flag.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1221: HDDS-1488. Scm cli command to start/stop replication manager.
bharatviswa504 commented on a change in pull request #1221: HDDS-1488. Scm cli
command to start/stop replication manager.
URL: https://github.com/apache/hadoop/pull/1221#discussion_r310406193
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java
##
@@ -469,6 +469,27 @@ public boolean forceExitSafeMode() throws IOException {
return scm.exitSafeMode();
}
+ @Override
+ public void startReplicationManager() {
+AUDIT.logWriteSuccess(buildAuditMessageForSuccess(
+SCMAction.START_REPLICATION_MANAGER, null));
+scm.getReplicationManager().start();
+ }
+
+ @Override
+ public void stopReplicationManager() {
+AUDIT.logWriteSuccess(buildAuditMessageForSuccess(
+SCMAction.STOP_REPLICATION_MANAGER, null));
+scm.getReplicationManager().stop();
Review comment:
This is true, even if we don't return status, as stop() is not a blocking
call.
Can we print some output message for the command for stop like we signalled
for stop, and if immediately status is called you can still see it as running.
(Or something more clearer, to let users know the behavior)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c…
steveloughran commented on issue #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c… URL: https://github.com/apache/hadoop/pull/1222#issuecomment-518018384 1. your IDE has gone overboard with reorganising the imports, which makes reviewing too hard. can you revert those changes so there's something more minimal. 1. If we can move all internal code off json serialization then there's no need to retain that class at all. it's private and not AFAIK used externally (it only got moved from hadoop-registry to hadoop-common with Hadoop 3.1, after all. >. And for hadoop-aws, maybe we can just move copy some code for JsonSerialization there, as the java sdk for aws depends on jackson so it is useless to purge the jackson dependency for it. AWS SDK is shaded; their problem. We need to get hadoop-aws off it, which is doable. Hadoop-registry is the origin of this code; need to review that to make sure the records persistet can be marshalled without problems For all the changes, I think we should add tests verifying consistent loading of existing classes first. That is: we save the current .json files as test resources and verify that they can be loaded with the new code. That's a good practise anyway, isn't it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c…
steveloughran commented on a change in pull request #1222: HADOOP-16488
Deprecated JsonSerialization and move it out of hadoop-c…
URL: https://github.com/apache/hadoop/pull/1222#discussion_r310394644
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
##
@@ -3718,52 +3709,42 @@ public static void dumpConfiguration(Configuration
config,
*/
public static void dumpConfiguration(Configuration config,
Writer out) throws IOException {
-JsonFactory dumpFactory = new JsonFactory();
-JsonGenerator dumpGenerator = dumpFactory.createGenerator(out);
-dumpGenerator.writeStartObject();
-dumpGenerator.writeFieldName("properties");
-dumpGenerator.writeStartArray();
-dumpGenerator.flush();
+JsonWriter jsonWriter = new JsonWriter(out);
+jsonWriter.beginObject().name("properties").beginArray().flush();
Review comment:
do we have tests for this writing of configs? If not, they're needed first
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c…
steveloughran commented on a change in pull request #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c… URL: https://github.com/apache/hadoop/pull/1222#discussion_r310394607 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java ## @@ -106,12 +106,6 @@ import org.w3c.dom.Document; import org.w3c.dom.Element; -import com.google.common.base.Preconditions; -import com.google.common.base.Strings; - -import static org.apache.commons.lang3.StringUtils.isBlank; -import static org.apache.commons.lang3.StringUtils.isNotBlank; - Review comment: your IDE is moving imports around which is shouldn't; this always makes backporting of patches to a key class harder. Can you leave out all changes to the imports except for those classes added/removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16485) Remove dependency on jackson
[ https://issues.apache.org/jira/browse/HADOOP-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899654#comment-16899654 ] Steve Loughran commented on HADOOP-16485: - It is tagged @Private/Evolving and mostly gets used for * conversion to from a map. This is something json can do. Uses, KMS, hadoop-azure, .. *in hadoop-aws ser/deser of the s3a committer .pendingset and _SUCCESS files. These (private) classes can be changed however we want; I'd prefer the persistence formats to stay the same but that's all. * hadoop-registry persistence of records in ZK. Again, whatever changes needed for gson are viable there. > Remove dependency on jackson > > > Key: HADOOP-16485 > URL: https://issues.apache.org/jira/browse/HADOOP-16485 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Priority: Major > > Looking at git history, there were 5 commits related to updating jackson > versions due to various CVEs since 2018. And it seems to get worse more > recently. > File this jira to discuss the possibility of removing jackson dependency once > for all. I see that jackson is deeply integrated into Hadoop codebase, so not > a trivial task. However, if Hadoop is forced to make a new set of releases > because of Jackson vulnerabilities, it may start to look not so costly. > At the very least, consider stripping jackson-databind coode, since that's > where the majority of CVEs come from. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1224: HDDS-1810. SCM command to Activate and Deactivate pipelines.
hadoop-yetus commented on issue #1224: HDDS-1810. SCM command to Activate and Deactivate pipelines. URL: https://github.com/apache/hadoop/pull/1224#issuecomment-518017189 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 76 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 84 | Maven dependency ordering for branch | | +1 | mvninstall | 617 | trunk passed | | +1 | compile | 367 | trunk passed | | +1 | checkstyle | 71 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 928 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 164 | trunk passed | | 0 | spotbugs | 430 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 630 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 30 | Maven dependency ordering for patch | | +1 | mvninstall | 560 | the patch passed | | +1 | compile | 369 | the patch passed | | +1 | cc | 369 | the patch passed | | +1 | javac | 369 | the patch passed | | +1 | checkstyle | 76 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 740 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 164 | the patch passed | | +1 | findbugs | 647 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 348 | hadoop-hdds in the patch passed. | | -1 | unit | 2120 | hadoop-ozone in the patch failed. | | +1 | asflicense | 43 | The patch does not generate ASF License warnings. | | | | 8203 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.om.TestScmSafeMode | | | hadoop.hdds.scm.pipeline.TestPipelineClose | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1224 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 9ca97659c82a 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 065cbc6 | | Default Java | 1.8.0_222 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/1/testReport/ | | Max. process+thread count | 4698 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/client hadoop-hdds/common hadoop-hdds/server-scm hadoop-hdds/tools hadoop-ozone/integration-test U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 opened a new pull request #1224: HDDS-1810. SCM command to Activate and Deactivate pipelines.
nandakumar131 opened a new pull request #1224: HDDS-1810. SCM command to Activate and Deactivate pipelines. URL: https://github.com/apache/hadoop/pull/1224 It will be useful to have scm command to temporarily deactivate and re-activate a pipeline. This will help us a lot in debugging a pipeline. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899603#comment-16899603
]
Hadoop QA commented on HADOOP-16112:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 10 new + 157 unchanged - 0 fixed = 167 total (was 157) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 30s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
8s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HADOOP-16112 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12976633/HADOOP-16112.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 29333c912cc2 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 065cbc6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16457/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16457/testReport/ |
| Max. process+thread count | 1361 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16457/console |
| Powered by | Apache
[GitHub] [hadoop] Hexiaoqiao commented on issue #1028: HDFS-14617 - Improve fsimage load time by writing sub-sections to the fsimage index
Hexiaoqiao commented on issue #1028: HDFS-14617 - Improve fsimage load time by writing sub-sections to the fsimage index URL: https://github.com/apache/hadoop/pull/1028#issuecomment-517991426 > And we should make sure oiv tool works with this change. We can file another jira to address the oiv issue. @jojochuang Thanks for your suggestions, current patch @sodonnel submitted seems not include OIV tools, And I agree that we should file another JIRA to trace it. I would like to file it, and help to trace it if @sodonnel have no times. FYI. Another side, I wonder if we should supply some other tools or compatible upgrade to cover the case when parallel loading fail, then fallback to old serial loading with new FSImage format. I do not doubt the parallel loading feature, just want to give more choice to users. Consider that when parallel loading failed due to some exception with new FSImage format, user could have no proper way to fallback even using OIV tools to transfer other format. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on a change in pull request #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c…
hadoop-yetus commented on a change in pull request #1222: HADOOP-16488
Deprecated JsonSerialization and move it out of hadoop-c…
URL: https://github.com/apache/hadoop/pull/1222#discussion_r310380320
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JsonSerialization.java
##
@@ -57,9 +57,13 @@
* {@code org.apache.hadoop.registry.client.binding.JsonSerDeser},
* which is now a subclass of this class.
* @param Type to marshal.
+ * @deprecated Avoid using this class any more, as jackson has
Review comment:
whitespace:end of line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c…
hadoop-yetus commented on issue #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c… URL: https://github.com/apache/hadoop/pull/1222#issuecomment-517990909 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 45 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 83 | Maven dependency ordering for branch | | +1 | mvninstall | 1068 | trunk passed | | +1 | compile | 1047 | trunk passed | | +1 | checkstyle | 148 | trunk passed | | +1 | mvnsite | 368 | trunk passed | | +1 | shadedclient | 1261 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 278 | trunk passed | | 0 | spotbugs | 55 | Used deprecated FindBugs config; considering switching to SpotBugs. | | 0 | findbugs | 24 | branch/hadoop-project no findbugs output file (findbugsXml.xml) | ||| _ Patch Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for patch | | +1 | mvninstall | 206 | the patch passed | | +1 | compile | 1015 | the patch passed | | -1 | javac | 1015 | root generated 13 new + 1482 unchanged - 0 fixed = 1495 total (was 1482) | | -0 | checkstyle | 133 | root: The patch generated 5 new + 358 unchanged - 3 fixed = 363 total (was 361) | | +1 | mvnsite | 287 | the patch passed | | -1 | whitespace | 0 | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 | xml | 3 | The patch has no ill-formed XML file. | | +1 | shadedclient | 614 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 236 | the patch passed | | 0 | findbugs | 23 | hadoop-project has no data from findbugs | ||| _ Other Tests _ | | +1 | unit | 24 | hadoop-project in the patch passed. | | -1 | unit | 478 | hadoop-common in the patch failed. | | -1 | unit | 122 | hadoop-kms in the patch failed. | | +1 | unit | 122 | hadoop-hdfs-client in the patch passed. | | +1 | unit | 331 | hadoop-mapreduce-client-core in the patch passed. | | +1 | unit | 301 | hadoop-aws in the patch passed. | | +1 | unit | 94 | hadoop-azure in the patch passed. | | +1 | asflicense | 41 | The patch does not generate ASF License warnings. | | | | 9178 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.security.token.delegation.web.TestWebDelegationToken | | | hadoop.crypto.key.kms.server.TestKMS | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1222/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1222 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | uname | Linux d84e585ad60a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 065cbc6 | | Default Java | 1.8.0_212 | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1222/1/artifact/out/diff-compile-javac-root.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1222/1/artifact/out/diff-checkstyle-root.txt | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1222/1/artifact/out/whitespace-eol.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1222/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1222/1/artifact/out/patch-unit-hadoop-common-project_hadoop-kms.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1222/1/testReport/ | | Max. process+thread count | 1640 (vs. ulimit of 5500) | | modules | C: hadoop-project hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1222/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service.
[jira] [Updated] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HADOOP-16112:
-
Attachment: HADOOP-16112.003.patch
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch,
> HADOOP-16112.003.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] leosunli opened a new pull request #1223: HADOOP-16112. Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
leosunli opened a new pull request #1223: HADOOP-16112. Delete the baseTrashPath's subDir leads to don't modify baseTrashPath URL: https://github.com/apache/hadoop/pull/1223 Signed-off-by: sunlisheng This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Apache9 commented on issue #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c…
Apache9 commented on issue #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c… URL: https://github.com/apache/hadoop/pull/1222#issuecomment-517981809 Ping @steveloughran @jojochuang . This is the first try. Not finished yet. The approach is to introduce a GsonSerialization, and use it to replace the JsonSerialization. And now the difficulty is that, we use the jackson annotation in many places, so I still need sometime to find out how to deal with these annotations. And for hadoop-aws, maybe we can just move copy some code for JsonSerialization there, as the java sdk for aws depends on jackson so it is useless to purge the jackson dependency for it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Apache9 opened a new pull request #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c…
Apache9 opened a new pull request #1222: HADOOP-16488 Deprecated JsonSerialization and move it out of hadoop-c… URL: https://github.com/apache/hadoop/pull/1222 …ommon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 commented on a change in pull request #1221: HDDS-1488. Scm cli command to start/stop replication manager.
nandakumar131 commented on a change in pull request #1221: HDDS-1488. Scm cli
command to start/stop replication manager.
URL: https://github.com/apache/hadoop/pull/1221#discussion_r310374189
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java
##
@@ -469,6 +469,27 @@ public boolean forceExitSafeMode() throws IOException {
return scm.exitSafeMode();
}
+ @Override
+ public void startReplicationManager() {
+AUDIT.logWriteSuccess(buildAuditMessageForSuccess(
+SCMAction.START_REPLICATION_MANAGER, null));
+scm.getReplicationManager().start();
+ }
+
+ @Override
+ public void stopReplicationManager() {
+AUDIT.logWriteSuccess(buildAuditMessageForSuccess(
+SCMAction.STOP_REPLICATION_MANAGER, null));
+scm.getReplicationManager().stop();
Review comment:
Stop is an idempotent call and also it's async, so getting status of
ReplicationManager immediately after calling stop might some time return true
(running).
> Like case calling stop on already stopped replication monitor, we can
print like Replication Monitor is not running
If we do this, we can end up in a minor inconsistency for a short period of
time. Where stop will say that ReplicationManager is not running, but the
status command will say that ReplicationManager is running.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16488) Deprecated JsonSerialization and move it out of hadoop-common
Duo Zhang created HADOOP-16488: -- Summary: Deprecated JsonSerialization and move it out of hadoop-common Key: HADOOP-16488 URL: https://issues.apache.org/jira/browse/HADOOP-16488 Project: Hadoop Common Issue Type: Sub-task Components: common Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16453) Update how exceptions are handled in NetUtils.java
[
https://issues.apache.org/jira/browse/HADOOP-16453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899570#comment-16899570
]
He Xiaoqiao commented on HADOOP-16453:
--
+1 for [^HADOOP-16453.002.patch] from my side. Thanks [~leosun08].
> Update how exceptions are handled in NetUtils.java
> --
>
> Key: HADOOP-16453
> URL: https://issues.apache.org/jira/browse/HADOOP-16453
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Minor
> Attachments: HADOOP-16453.001.patch, HADOOP-16453.002.patch
>
>
> When there is no String Constructor for the exception, we Log a Trace
> Message. Given that log and throw is not a very good approach I think the
> right thing would be to just not log it at all as HADOOP-16431.
> {code:java}
> private static T wrapWithMessage(
> T exception, String msg) throws T {
> Class clazz = exception.getClass();
> try {
> Constructor ctor =
> clazz.getConstructor(String.class);
> Throwable t = ctor.newInstance(msg);
> return (T)(t.initCause(exception));
> } catch (Throwable e) {
> LOG.trace("Unable to wrap exception of type {}: it has no (String) "
> + "constructor", clazz, e);
> throw exception;
> }
> }
> {code}
> *exception stack：*
> {code:java}
> 19/07/12 11:23:45 INFO mapreduce.JobSubmitter: Executing with tokens: [Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:azorprc-xiaomi, Ident: (token for
> sql_prc: HDFS_DELEGATION_TOKEN owner=sql_prc/hadoop@XIAOMI.HADOOP,
> renewer=yarn_prc, realUser=, issueDate=1562901814007, maxDate=1594437814007,
> sequenceNumber=3349939, masterKeyId=1400)]
> 19/07/12 11:23:46 TRACE net.NetUtils: Unable to wrap exception of type class
> java.nio.channels.ClosedByInterruptException: it has no (String) constructor
> java.lang.NoSuchMethodException:
> java.nio.channels.ClosedByInterruptException.(java.lang.String)
> at java.lang.Class.getConstructor0(Class.java:3082)
> at java.lang.Class.getConstructor(Class.java:1825)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:830)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:806)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1559)
> at org.apache.hadoop.ipc.Client.call(Client.java:1501)
> at org.apache.hadoop.ipc.Client.call(Client.java:1411)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:949)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.RequestHedgingProxyProvider$RequestHedgingInvocationHandler$1.call(RequestHedgingProxyProvider.java:143)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 19/07/12 11:23:46 INFO Configuration.deprecation: No unit for
> dfs.client.datanode-restart.timeout(30) assuming SECONDS
> 19/07/12 11:23:46 INFO Configuration.deprecation: No unit for
> dfs.client.datanode-restart.timeout(30) assuming SECONDS
> 19/07/12 11:23:46 INFO Configuration.deprecation: No unit for
> dfs.client.datanode-restart.timeout(30) assuming SECONDS
> 19/07/12 11:23:46 INFO Configuration.deprecation: No unit for
> dfs.client.datanode-restart.timeout(30) assuming SECONDS
> 19/07/12 11:23:46 INFO Configuration.deprecation: No unit for
> dfs.client.datanode-restart.timeout(30) assuming SECONDS
> 19/07/12 11:23:46 INFO Configuration.deprecation: No unit for
> dfs.client.datanode-restart.timeout(30) assuming SECONDS
> 19/07/12 11:23:46 WARN ipc.Client: Exception encountered while connecting to
> the server : java.io.InterruptedIOException: Interrupted while waiting for IO
> on channel java.nio.channels.SocketChannel[connected
> local=/10.118.30.48:34324 remote=/10.69.11.137:11200]. 6 millis timeout
> left.
> 19/07/12 11:23:48 INFO conf.Configuration: resource-types.xml not found
> 19/07/12 11:23:48 INFO resource.ResourceUtils: Unable to find
> 'resource-types.xml'.
>
