[jira] [Work logged] (HDDS-2132) TestKeyValueContainer is failing
[
https://issues.apache.org/jira/browse/HDDS-2132?focusedWorklogId=313511=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313511
]
ASF GitHub Bot logged work on HDDS-2132:
Author: ASF GitHub Bot
Created on: 17/Sep/19 05:58
Start Date: 17/Sep/19 05:58
Worklog Time Spent: 10m
Work Description: adoroszlai commented on issue #1457: HDDS-2132.
TestKeyValueContainer is failing
URL: https://github.com/apache/hadoop/pull/1457#issuecomment-532071738
@bshashikant please review
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313511)
Time Spent: 0.5h (was: 20m)
> TestKeyValueContainer is failing
>
>
> Key: HDDS-2132
> URL: https://issues.apache.org/jira/browse/HDDS-2132
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Doroszlai, Attila
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {{TestKeyValueContainer}} is failing with the following exception
> {noformat}
> [ERROR]
> testContainerImportExport(org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer)
> Time elapsed: 0.173 s <<< ERROR!
> java.lang.NullPointerException
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
> at
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.parseKVContainerData(KeyValueContainerUtil.java:201)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.importContainerData(KeyValueContainer.java:500)
> at
> org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer.testContainerImportExport(TestKeyValueContainer.java:235)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Work logged] (HDDS-2016) Add option to enforce gdpr in Bucket Create command
[ https://issues.apache.org/jira/browse/HDDS-2016?focusedWorklogId=313507=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313507 ] ASF GitHub Bot logged work on HDDS-2016: Author: ASF GitHub Bot Created on: 17/Sep/19 05:44 Start Date: 17/Sep/19 05:44 Worklog Time Spent: 10m Work Description: dineshchitlangia commented on pull request #1458: HDDS-2016. Add option to enforce gdpr in Bucket Create command. URL: https://github.com/apache/hadoop/pull/1458 Verified by running in a local cluster as the previous patches for this feature covers the tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313507) Remaining Estimate: 0h Time Spent: 10m > Add option to enforce gdpr in Bucket Create command > --- > > Key: HDDS-2016 > URL: https://issues.apache.org/jira/browse/HDDS-2016 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > e2e flow where user can enforce GDPR for a bucket during creation only. > Add/update audit logs as this will be a useful action for compliance purpose. > Add docs to show usage. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2016) Add option to enforce gdpr in Bucket Create command
[ https://issues.apache.org/jira/browse/HDDS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2016: - Labels: pull-request-available (was: ) > Add option to enforce gdpr in Bucket Create command > --- > > Key: HDDS-2016 > URL: https://issues.apache.org/jira/browse/HDDS-2016 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > Labels: pull-request-available > > e2e flow where user can enforce GDPR for a bucket during creation only. > Add/update audit logs as this will be a useful action for compliance purpose. > Add docs to show usage. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2016) Add option to enforce gdpr in Bucket Create command
[ https://issues.apache.org/jira/browse/HDDS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HDDS-2016: Description: e2e flow where user can enforce GDPR for a bucket during creation only. Add/update audit logs as this will be a useful action for compliance purpose. Add docs to show usage. was: e2e flow where user can enable GDPR for a bucket during creation only. Add/update audit logs as this will be a useful action for compliance purpose. Add docs to show usage. > Add option to enforce gdpr in Bucket Create command > --- > > Key: HDDS-2016 > URL: https://issues.apache.org/jira/browse/HDDS-2016 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > > e2e flow where user can enforce GDPR for a bucket during creation only. > Add/update audit logs as this will be a useful action for compliance purpose. > Add docs to show usage. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2016) Add option to enforce gdpr in Bucket Create command
[ https://issues.apache.org/jira/browse/HDDS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HDDS-2016: Summary: Add option to enforce gdpr in Bucket Create command (was: Add option to enable gdpr in Bucket Create command) > Add option to enforce gdpr in Bucket Create command > --- > > Key: HDDS-2016 > URL: https://issues.apache.org/jira/browse/HDDS-2016 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > > e2e flow where user can enable GDPR for a bucket during creation only. > Add/update audit logs as this will be a useful action for compliance purpose. > Add docs to show usage. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2016) Add option to enable gdpr in Bucket Create command
[ https://issues.apache.org/jira/browse/HDDS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HDDS-2016: Description: e2e flow where user can enable GDPR for a bucket during creation only. Add/update audit logs as this will be a useful action for compliance purpose. Add docs to show usage. was: e2e flow where user can enable GDPR for a bucket during creation only. Add/update audit logs as this will be a useful action for compliance purpose. > Add option to enable gdpr in Bucket Create command > -- > > Key: HDDS-2016 > URL: https://issues.apache.org/jira/browse/HDDS-2016 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > > e2e flow where user can enable GDPR for a bucket during creation only. > Add/update audit logs as this will be a useful action for compliance purpose. > Add docs to show usage. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2098) Ozone shell command prints out ERROR when the log4j file is not present.
[
https://issues.apache.org/jira/browse/HDDS-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-2098:
-
Status: Reopened (was: Reopened)
> Ozone shell command prints out ERROR when the log4j file is not present.
>
>
> Key: HDDS-2098
> URL: https://issues.apache.org/jira/browse/HDDS-2098
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> *Exception Trace*
> {code}
> log4j:ERROR Could not read configuration file from URL
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> java.io.FileNotFoundException: /etc/ozone/conf/ozone-shell-log4j.properties
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.(FileInputStream.java:138)
> at java.io.FileInputStream.(FileInputStream.java:93)
> at
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66)
> at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
> at
> org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
> at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
> at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
> at org.apache.hadoop.ozone.web.ozShell.Shell.(Shell.java:35)
> log4j:ERROR Ignoring configuration file
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> log4j:WARN No appenders could be found for logger
> (io.jaegertracing.thrift.internal.senders.ThriftSenderFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> {
> "metadata" : { },
> "name" : "vol-test-putfile-1567740142",
> "admin" : "root",
> "owner" : "root",
> "creationTime" : 1567740146501,
> "acls" : [ {
> "type" : "USER",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ],
> "quota" : 1152921504606846976
> }
> {code}
> *Fix*
> When a log4j file is not present, the default should be console.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2098) Ozone shell command prints out ERROR when the log4j file is not present.
[
https://issues.apache.org/jira/browse/HDDS-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham resolved HDDS-2098.
--
Resolution: Fixed
> Ozone shell command prints out ERROR when the log4j file is not present.
>
>
> Key: HDDS-2098
> URL: https://issues.apache.org/jira/browse/HDDS-2098
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> *Exception Trace*
> {code}
> log4j:ERROR Could not read configuration file from URL
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> java.io.FileNotFoundException: /etc/ozone/conf/ozone-shell-log4j.properties
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.(FileInputStream.java:138)
> at java.io.FileInputStream.(FileInputStream.java:93)
> at
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66)
> at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
> at
> org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
> at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
> at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
> at org.apache.hadoop.ozone.web.ozShell.Shell.(Shell.java:35)
> log4j:ERROR Ignoring configuration file
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> log4j:WARN No appenders could be found for logger
> (io.jaegertracing.thrift.internal.senders.ThriftSenderFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> {
> "metadata" : { },
> "name" : "vol-test-putfile-1567740142",
> "admin" : "root",
> "owner" : "root",
> "creationTime" : 1567740146501,
> "acls" : [ {
> "type" : "USER",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ],
> "quota" : 1152921504606846976
> }
> {code}
> *Fix*
> When a log4j file is not present, the default should be console.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2098) Ozone shell command prints out ERROR when the log4j file is not present.
[
https://issues.apache.org/jira/browse/HDDS-2098?focusedWorklogId=313504=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313504
]
ASF GitHub Bot logged work on HDDS-2098:
Author: ASF GitHub Bot
Created on: 17/Sep/19 05:36
Start Date: 17/Sep/19 05:36
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1411:
HDDS-2098 : Ozone shell command prints out ERROR when the log4j file …
URL: https://github.com/apache/hadoop/pull/1411
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313504)
Time Spent: 2.5h (was: 2h 20m)
> Ozone shell command prints out ERROR when the log4j file is not present.
>
>
> Key: HDDS-2098
> URL: https://issues.apache.org/jira/browse/HDDS-2098
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> *Exception Trace*
> {code}
> log4j:ERROR Could not read configuration file from URL
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> java.io.FileNotFoundException: /etc/ozone/conf/ozone-shell-log4j.properties
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.(FileInputStream.java:138)
> at java.io.FileInputStream.(FileInputStream.java:93)
> at
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66)
> at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
> at
> org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
> at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
> at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
> at org.apache.hadoop.ozone.web.ozShell.Shell.(Shell.java:35)
> log4j:ERROR Ignoring configuration file
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> log4j:WARN No appenders could be found for logger
> (io.jaegertracing.thrift.internal.senders.ThriftSenderFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> {
> "metadata" : { },
> "name" : "vol-test-putfile-1567740142",
> "admin" : "root",
> "owner" : "root",
> "creationTime" : 1567740146501,
> "acls" : [ {
> "type" : "USER",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ],
> "quota" : 1152921504606846976
> }
> {code}
> *Fix*
> When a log4j file is not present, the default should be console.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14768) In some cases, erasure blocks are corruption when they are reconstruct.
[
https://issues.apache.org/jira/browse/HDFS-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
guojh updated HDFS-14768:
-
Attachment: (was: HDFS-14768.0001.patch)
> In some cases, erasure blocks are corruption when they are reconstruct.
>
>
> Key: HDFS-14768
> URL: https://issues.apache.org/jira/browse/HDFS-14768
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding, hdfs, namenode
>Affects Versions: 3.0.2
>Reporter: guojh
>Assignee: guojh
>Priority: Major
> Labels: patch
> Fix For: 3.3.0
>
> Attachments: 1568275810244.jpg, 1568276338275.jpg,
> HDFS-14768.000.patch, HDFS-14768.001.patch, guojh_UT_after_deomission.txt,
> guojh_UT_before_deomission.txt, zhaoyiming_UT_after_deomission.txt,
> zhaoyiming_UT_beofre_deomission.txt
>
>
> Policy is RS-6-3-1024K, version is hadoop 3.0.2;
> We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
> index[3,4], increase the index 6 datanode's
> pendingReplicationWithoutTargets that make it large than
> replicationStreamsHardLimit(we set 14). Then, After the method
> chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
> [0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
> In method scheduleReconstruction of BlockManager, the additionalReplRequired
> is 9 - 7 = 2. After Namenode choose two target Datanode, will assign a
> erasureCode task to target datanode.
> When datanode get the task will build targetIndices from liveBlockIndices
> and target length. the code is blow.
> {code:java}
> // code placeholder
> targetIndices = new short[targets.length];
> private void initTargetIndices() {
> BitSet bitset = reconstructor.getLiveBitSet();
> int m = 0; hasValidTargets = false;
> for (int i = 0; i < dataBlkNum + parityBlkNum; i++) {
> if (!bitset.get) {
> if (reconstructor.getBlockLen > 0) {
> if (m < targets.length) {
> targetIndices[m++] = (short)i;
> hasValidTargets = true;
> }
> }
> }
> }
> {code}
> targetIndices[0]=6, and targetIndices[1] is aways 0 from initial value.
> The StripedReader is aways create reader from first 6 index block, and is
> [0,1,2,3,4,5]
> Use the index [0,1,2,3,4,5] to build target index[6,0] will trigger the isal
> bug. the block index6's data is corruption(all data is zero).
> I write a unit test can stabilize repreduce.
> {code:java}
> // code placeholder
> private int replicationStreamsHardLimit =
> DFSConfigKeys.DFS_NAMENODE_REPLICATION_STREAMS_HARD_LIMIT_DEFAULT;
> numDNs = dataBlocks + parityBlocks + 10;
> @Test(timeout = 24)
> public void testFileDecommission() throws Exception {
> LOG.info("Starting test testFileDecommission");
> final Path ecFile = new Path(ecDir, "testFileDecommission");
> int writeBytes = cellSize * dataBlocks;
> writeStripedFile(dfs, ecFile, writeBytes);
> Assert.assertEquals(0, bm.numOfUnderReplicatedBlocks());
> FileChecksum fileChecksum1 = dfs.getFileChecksum(ecFile, writeBytes);
> final INodeFile fileNode = cluster.getNamesystem().getFSDirectory()
> .getINode4Write(ecFile.toString()).asFile();
> LocatedBlocks locatedBlocks =
> StripedFileTestUtil.getLocatedBlocks(ecFile, dfs);
> LocatedBlock lb = dfs.getClient().getLocatedBlocks(ecFile.toString(), 0)
> .get(0);
> DatanodeInfo[] dnLocs = lb.getLocations();
> LocatedStripedBlock lastBlock =
> (LocatedStripedBlock)locatedBlocks.getLastLocatedBlock();
> DatanodeInfo[] storageInfos = lastBlock.getLocations();
> //
> DatanodeDescriptor datanodeDescriptor =
> cluster.getNameNode().getNamesystem()
>
> .getBlockManager().getDatanodeManager().getDatanode(storageInfos[6].getDatanodeUuid());
> BlockInfo firstBlock = fileNode.getBlocks()[0];
> DatanodeStorageInfo[] dStorageInfos = bm.getStorages(firstBlock);
> // the first heartbeat will consume 3 replica tasks
> for (int i = 0; i <= replicationStreamsHardLimit + 3; i++) {
> BlockManagerTestUtil.addBlockToBeReplicated(datanodeDescriptor, new
> Block(i),
> new DatanodeStorageInfo[]{dStorageInfos[0]});
> }
> assertEquals(dataBlocks + parityBlocks, dnLocs.length);
> int[] decommNodeIndex = {3, 4};
> final List decommisionNodes = new ArrayList();
> // add the node which will be decommissioning
> decommisionNodes.add(dnLocs[decommNodeIndex[0]]);
> decommisionNodes.add(dnLocs[decommNodeIndex[1]]);
> decommissionNode(0, decommisionNodes, AdminStates.DECOMMISSIONED);
> assertEquals(decommisionNodes.size(), fsn.getNumDecomLiveDataNodes());
> bm.getDatanodeManager().removeDatanode(datanodeDescriptor);
> //assertNull(checkFile(dfs, ecFile, 9, decommisionNodes, numDNs));
> // Ensure decommissioned datanode is not
[jira] [Updated] (HDFS-14768) In some cases, erasure blocks are corruption when they are reconstruct.
[
https://issues.apache.org/jira/browse/HDFS-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
guojh updated HDFS-14768:
-
Attachment: HDFS-14768.001.patch
> In some cases, erasure blocks are corruption when they are reconstruct.
>
>
> Key: HDFS-14768
> URL: https://issues.apache.org/jira/browse/HDFS-14768
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding, hdfs, namenode
>Affects Versions: 3.0.2
>Reporter: guojh
>Assignee: guojh
>Priority: Major
> Labels: patch
> Fix For: 3.3.0
>
> Attachments: 1568275810244.jpg, 1568276338275.jpg,
> HDFS-14768.000.patch, HDFS-14768.001.patch, guojh_UT_after_deomission.txt,
> guojh_UT_before_deomission.txt, zhaoyiming_UT_after_deomission.txt,
> zhaoyiming_UT_beofre_deomission.txt
>
>
> Policy is RS-6-3-1024K, version is hadoop 3.0.2;
> We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
> index[3,4], increase the index 6 datanode's
> pendingReplicationWithoutTargets that make it large than
> replicationStreamsHardLimit(we set 14). Then, After the method
> chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
> [0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
> In method scheduleReconstruction of BlockManager, the additionalReplRequired
> is 9 - 7 = 2. After Namenode choose two target Datanode, will assign a
> erasureCode task to target datanode.
> When datanode get the task will build targetIndices from liveBlockIndices
> and target length. the code is blow.
> {code:java}
> // code placeholder
> targetIndices = new short[targets.length];
> private void initTargetIndices() {
> BitSet bitset = reconstructor.getLiveBitSet();
> int m = 0; hasValidTargets = false;
> for (int i = 0; i < dataBlkNum + parityBlkNum; i++) {
> if (!bitset.get) {
> if (reconstructor.getBlockLen > 0) {
> if (m < targets.length) {
> targetIndices[m++] = (short)i;
> hasValidTargets = true;
> }
> }
> }
> }
> {code}
> targetIndices[0]=6, and targetIndices[1] is aways 0 from initial value.
> The StripedReader is aways create reader from first 6 index block, and is
> [0,1,2,3,4,5]
> Use the index [0,1,2,3,4,5] to build target index[6,0] will trigger the isal
> bug. the block index6's data is corruption(all data is zero).
> I write a unit test can stabilize repreduce.
> {code:java}
> // code placeholder
> private int replicationStreamsHardLimit =
> DFSConfigKeys.DFS_NAMENODE_REPLICATION_STREAMS_HARD_LIMIT_DEFAULT;
> numDNs = dataBlocks + parityBlocks + 10;
> @Test(timeout = 24)
> public void testFileDecommission() throws Exception {
> LOG.info("Starting test testFileDecommission");
> final Path ecFile = new Path(ecDir, "testFileDecommission");
> int writeBytes = cellSize * dataBlocks;
> writeStripedFile(dfs, ecFile, writeBytes);
> Assert.assertEquals(0, bm.numOfUnderReplicatedBlocks());
> FileChecksum fileChecksum1 = dfs.getFileChecksum(ecFile, writeBytes);
> final INodeFile fileNode = cluster.getNamesystem().getFSDirectory()
> .getINode4Write(ecFile.toString()).asFile();
> LocatedBlocks locatedBlocks =
> StripedFileTestUtil.getLocatedBlocks(ecFile, dfs);
> LocatedBlock lb = dfs.getClient().getLocatedBlocks(ecFile.toString(), 0)
> .get(0);
> DatanodeInfo[] dnLocs = lb.getLocations();
> LocatedStripedBlock lastBlock =
> (LocatedStripedBlock)locatedBlocks.getLastLocatedBlock();
> DatanodeInfo[] storageInfos = lastBlock.getLocations();
> //
> DatanodeDescriptor datanodeDescriptor =
> cluster.getNameNode().getNamesystem()
>
> .getBlockManager().getDatanodeManager().getDatanode(storageInfos[6].getDatanodeUuid());
> BlockInfo firstBlock = fileNode.getBlocks()[0];
> DatanodeStorageInfo[] dStorageInfos = bm.getStorages(firstBlock);
> // the first heartbeat will consume 3 replica tasks
> for (int i = 0; i <= replicationStreamsHardLimit + 3; i++) {
> BlockManagerTestUtil.addBlockToBeReplicated(datanodeDescriptor, new
> Block(i),
> new DatanodeStorageInfo[]{dStorageInfos[0]});
> }
> assertEquals(dataBlocks + parityBlocks, dnLocs.length);
> int[] decommNodeIndex = {3, 4};
> final List decommisionNodes = new ArrayList();
> // add the node which will be decommissioning
> decommisionNodes.add(dnLocs[decommNodeIndex[0]]);
> decommisionNodes.add(dnLocs[decommNodeIndex[1]]);
> decommissionNode(0, decommisionNodes, AdminStates.DECOMMISSIONED);
> assertEquals(decommisionNodes.size(), fsn.getNumDecomLiveDataNodes());
> bm.getDatanodeManager().removeDatanode(datanodeDescriptor);
> //assertNull(checkFile(dfs, ecFile, 9, decommisionNodes, numDNs));
> // Ensure decommissioned datanode is not
[jira] [Commented] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[
https://issues.apache.org/jira/browse/HDDS-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931105#comment-16931105
]
Mukul Kumar Singh commented on HDDS-2114:
-
The build works out locally and hence committed this to trunk
> Rename does not preserve non-explicitly created interim directories
> ---
>
> Key: HDDS-2114
> URL: https://issues.apache.org/jira/browse/HDDS-2114
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Lokesh Jain
>Priority: Critical
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Attachments: demonstrative_test.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> I am attaching a patch that adds a test that demonstrates the problem.
> The scenario is coming from the way how Hive implements acid transactions
> with the ORC table format, but the test is redacted to the simplest possible
> code that reproduces the issue.
> The scenario:
> * Given a 3 level directory structure, where the top level directory was
> explicitly created, and the interim directory is implicitly created (for
> example either by creating a file with create("/top/interim/file") or by
> creating a directory with mkdirs("top/interim/dir"))
> * When the leaf is moved out from the implicitly created directory making
> this directory an empty directory
> * Then a FileNotFoundException is thrown when getFileStatus or listStatus is
> called on the interim directory.
> The expected behaviour:
> after the directory is becoming empty, the directory should still be part of
> the file system, moreover an empty FileStatus array should be returned when
> listStatus is called on it, and also a valid FileStatus object should be
> returned when getFileStatus is called on it.
>
>
> As this issue is present with Hive, and as this is how a FileSystem is
> expected to work this seems to be an at least critical issue as I see, please
> feel free to change the priority if needed.
> Also please note that, if the interim directory is explicitly created with
> mkdirs("top/interim") before creating the leaf, then the issue does not
> appear.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[
https://issues.apache.org/jira/browse/HDDS-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDDS-2114:
Fix Version/s: 0.5.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the contribution [~ljain] and [~pifta]. I have committed this to
trunk.
> Rename does not preserve non-explicitly created interim directories
> ---
>
> Key: HDDS-2114
> URL: https://issues.apache.org/jira/browse/HDDS-2114
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Lokesh Jain
>Priority: Critical
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Attachments: demonstrative_test.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> I am attaching a patch that adds a test that demonstrates the problem.
> The scenario is coming from the way how Hive implements acid transactions
> with the ORC table format, but the test is redacted to the simplest possible
> code that reproduces the issue.
> The scenario:
> * Given a 3 level directory structure, where the top level directory was
> explicitly created, and the interim directory is implicitly created (for
> example either by creating a file with create("/top/interim/file") or by
> creating a directory with mkdirs("top/interim/dir"))
> * When the leaf is moved out from the implicitly created directory making
> this directory an empty directory
> * Then a FileNotFoundException is thrown when getFileStatus or listStatus is
> called on the interim directory.
> The expected behaviour:
> after the directory is becoming empty, the directory should still be part of
> the file system, moreover an empty FileStatus array should be returned when
> listStatus is called on it, and also a valid FileStatus object should be
> returned when getFileStatus is called on it.
>
>
> As this issue is present with Hive, and as this is how a FileSystem is
> expected to work this seems to be an at least critical issue as I see, please
> feel free to change the priority if needed.
> Also please note that, if the interim directory is explicitly created with
> mkdirs("top/interim") before creating the leaf, then the issue does not
> appear.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2098) Ozone shell command prints out ERROR when the log4j file is not present.
[
https://issues.apache.org/jira/browse/HDDS-2098?focusedWorklogId=313496=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313496
]
ASF GitHub Bot logged work on HDDS-2098:
Author: ASF GitHub Bot
Created on: 17/Sep/19 05:14
Start Date: 17/Sep/19 05:14
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on issue #1411: HDDS-2098 : Ozone
shell command prints out ERROR when the log4j file …
URL: https://github.com/apache/hadoop/pull/1411#issuecomment-529639718
> I have a question
> During ozone tarball build, we do copy ozone-shell-log4j.properties to
etc/hadoop (like we copy log4.properties then why do we see this error or
something need to be fixed in copying this script?
>
>
https://github.com/apache/hadoop/blob/trunk/hadoop-ozone/dist/dev-support/bin/dist-layout-stitching#L95
Yes, while starting ozone from snapshot tar ball, it works perfectly.
However, when Ozone is deployed through a cluster management tool like Cloudera
Manager, the log4j properties may not be individually configurable. We may have
to rely on a default log4.properties. In that case, printing a
FileNotFoundException for ozone shell commands is something we can avoid.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313496)
Time Spent: 2h 20m (was: 2h 10m)
> Ozone shell command prints out ERROR when the log4j file is not present.
>
>
> Key: HDDS-2098
> URL: https://issues.apache.org/jira/browse/HDDS-2098
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> *Exception Trace*
> {code}
> log4j:ERROR Could not read configuration file from URL
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> java.io.FileNotFoundException: /etc/ozone/conf/ozone-shell-log4j.properties
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.(FileInputStream.java:138)
> at java.io.FileInputStream.(FileInputStream.java:93)
> at
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66)
> at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
> at
> org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
> at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
> at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
> at org.apache.hadoop.ozone.web.ozShell.Shell.(Shell.java:35)
> log4j:ERROR Ignoring configuration file
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> log4j:WARN No appenders could be found for logger
> (io.jaegertracing.thrift.internal.senders.ThriftSenderFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> {
> "metadata" : { },
> "name" : "vol-test-putfile-1567740142",
> "admin" : "root",
> "owner" : "root",
> "creationTime" : 1567740146501,
> "acls" : [ {
> "type" : "USER",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ],
> "quota" : 1152921504606846976
> }
> {code}
> *Fix*
> When a log4j file is not present, the default should be console.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (HDDS-2098) Ozone shell command prints out ERROR when the log4j file is not present.
[
https://issues.apache.org/jira/browse/HDDS-2098?focusedWorklogId=313494=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313494
]
ASF GitHub Bot logged work on HDDS-2098:
Author: ASF GitHub Bot
Created on: 17/Sep/19 05:12
Start Date: 17/Sep/19 05:12
Worklog Time Spent: 10m
Work Description: adoroszlai commented on issue #1411: HDDS-2098 : Ozone
shell command prints out ERROR when the log4j file …
URL: https://github.com/apache/hadoop/pull/1411#issuecomment-532061447
Thanks for fixing this @avijayanhwx.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313494)
Time Spent: 2h 10m (was: 2h)
> Ozone shell command prints out ERROR when the log4j file is not present.
>
>
> Key: HDDS-2098
> URL: https://issues.apache.org/jira/browse/HDDS-2098
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> *Exception Trace*
> {code}
> log4j:ERROR Could not read configuration file from URL
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> java.io.FileNotFoundException: /etc/ozone/conf/ozone-shell-log4j.properties
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.(FileInputStream.java:138)
> at java.io.FileInputStream.(FileInputStream.java:93)
> at
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66)
> at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
> at
> org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
> at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
> at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
> at org.apache.hadoop.ozone.web.ozShell.Shell.(Shell.java:35)
> log4j:ERROR Ignoring configuration file
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> log4j:WARN No appenders could be found for logger
> (io.jaegertracing.thrift.internal.senders.ThriftSenderFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> {
> "metadata" : { },
> "name" : "vol-test-putfile-1567740142",
> "admin" : "root",
> "owner" : "root",
> "creationTime" : 1567740146501,
> "acls" : [ {
> "type" : "USER",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ],
> "quota" : 1152921504606846976
> }
> {code}
> *Fix*
> When a log4j file is not present, the default should be console.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDDS-2098) Ozone shell command prints out ERROR when the log4j file is not present.
[
https://issues.apache.org/jira/browse/HDDS-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan reopened HDDS-2098:
-
> Ozone shell command prints out ERROR when the log4j file is not present.
>
>
> Key: HDDS-2098
> URL: https://issues.apache.org/jira/browse/HDDS-2098
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> *Exception Trace*
> {code}
> log4j:ERROR Could not read configuration file from URL
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> java.io.FileNotFoundException: /etc/ozone/conf/ozone-shell-log4j.properties
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.(FileInputStream.java:138)
> at java.io.FileInputStream.(FileInputStream.java:93)
> at
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66)
> at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
> at
> org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
> at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
> at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
> at org.apache.hadoop.ozone.web.ozShell.Shell.(Shell.java:35)
> log4j:ERROR Ignoring configuration file
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> log4j:WARN No appenders could be found for logger
> (io.jaegertracing.thrift.internal.senders.ThriftSenderFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> {
> "metadata" : { },
> "name" : "vol-test-putfile-1567740142",
> "admin" : "root",
> "owner" : "root",
> "creationTime" : 1567740146501,
> "acls" : [ {
> "type" : "USER",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ],
> "quota" : 1152921504606846976
> }
> {code}
> *Fix*
> When a log4j file is not present, the default should be console.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2098) Ozone shell command prints out ERROR when the log4j file is not present.
[
https://issues.apache.org/jira/browse/HDDS-2098?focusedWorklogId=313491=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313491
]
ASF GitHub Bot logged work on HDDS-2098:
Author: ASF GitHub Bot
Created on: 17/Sep/19 05:10
Start Date: 17/Sep/19 05:10
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on pull request #1411: HDDS-2098
: Ozone shell command prints out ERROR when the log4j file …
URL: https://github.com/apache/hadoop/pull/1411
…is not present.
Manually tested change on cluster.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313491)
Time Spent: 1h 50m (was: 1h 40m)
> Ozone shell command prints out ERROR when the log4j file is not present.
>
>
> Key: HDDS-2098
> URL: https://issues.apache.org/jira/browse/HDDS-2098
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> *Exception Trace*
> {code}
> log4j:ERROR Could not read configuration file from URL
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> java.io.FileNotFoundException: /etc/ozone/conf/ozone-shell-log4j.properties
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.(FileInputStream.java:138)
> at java.io.FileInputStream.(FileInputStream.java:93)
> at
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66)
> at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
> at
> org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
> at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
> at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
> at org.apache.hadoop.ozone.web.ozShell.Shell.(Shell.java:35)
> log4j:ERROR Ignoring configuration file
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> log4j:WARN No appenders could be found for logger
> (io.jaegertracing.thrift.internal.senders.ThriftSenderFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> {
> "metadata" : { },
> "name" : "vol-test-putfile-1567740142",
> "admin" : "root",
> "owner" : "root",
> "creationTime" : 1567740146501,
> "acls" : [ {
> "type" : "USER",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ],
> "quota" : 1152921504606846976
> }
> {code}
> *Fix*
> When a log4j file is not present, the default should be console.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2098) Ozone shell command prints out ERROR when the log4j file is not present.
[
https://issues.apache.org/jira/browse/HDDS-2098?focusedWorklogId=313492=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313492
]
ASF GitHub Bot logged work on HDDS-2098:
Author: ASF GitHub Bot
Created on: 17/Sep/19 05:10
Start Date: 17/Sep/19 05:10
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on issue #1411: HDDS-2098 : Ozone
shell command prints out ERROR when the log4j file …
URL: https://github.com/apache/hadoop/pull/1411#issuecomment-531937463
Closing this out since we can handle it from the cluster management tool.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313492)
Time Spent: 2h (was: 1h 50m)
> Ozone shell command prints out ERROR when the log4j file is not present.
>
>
> Key: HDDS-2098
> URL: https://issues.apache.org/jira/browse/HDDS-2098
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> *Exception Trace*
> {code}
> log4j:ERROR Could not read configuration file from URL
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> java.io.FileNotFoundException: /etc/ozone/conf/ozone-shell-log4j.properties
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.(FileInputStream.java:138)
> at java.io.FileInputStream.(FileInputStream.java:93)
> at
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66)
> at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
> at
> org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
> at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
> at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
> at org.apache.hadoop.ozone.web.ozShell.Shell.(Shell.java:35)
> log4j:ERROR Ignoring configuration file
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> log4j:WARN No appenders could be found for logger
> (io.jaegertracing.thrift.internal.senders.ThriftSenderFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> {
> "metadata" : { },
> "name" : "vol-test-putfile-1567740142",
> "admin" : "root",
> "owner" : "root",
> "creationTime" : 1567740146501,
> "acls" : [ {
> "type" : "USER",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ],
> "quota" : 1152921504606846976
> }
> {code}
> *Fix*
> When a log4j file is not present, the default should be console.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDDS-2098) Ozone shell command prints out ERROR when the log4j file is not present.
[
https://issues.apache.org/jira/browse/HDDS-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-2098:
Comment: was deleted
(was: This is no longer needed since we can handle it in the cluster management
tool (CM, Ambari etc) level. )
> Ozone shell command prints out ERROR when the log4j file is not present.
>
>
> Key: HDDS-2098
> URL: https://issues.apache.org/jira/browse/HDDS-2098
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> *Exception Trace*
> {code}
> log4j:ERROR Could not read configuration file from URL
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> java.io.FileNotFoundException: /etc/ozone/conf/ozone-shell-log4j.properties
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.(FileInputStream.java:138)
> at java.io.FileInputStream.(FileInputStream.java:93)
> at
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66)
> at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
> at
> org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
> at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
> at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
> at org.apache.hadoop.ozone.web.ozShell.Shell.(Shell.java:35)
> log4j:ERROR Ignoring configuration file
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> log4j:WARN No appenders could be found for logger
> (io.jaegertracing.thrift.internal.senders.ThriftSenderFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> {
> "metadata" : { },
> "name" : "vol-test-putfile-1567740142",
> "admin" : "root",
> "owner" : "root",
> "creationTime" : 1567740146501,
> "acls" : [ {
> "type" : "USER",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ],
> "quota" : 1152921504606846976
> }
> {code}
> *Fix*
> When a log4j file is not present, the default should be console.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14768) In some cases, erasure blocks are corruption when they are reconstruct.
[
https://issues.apache.org/jira/browse/HDFS-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931098#comment-16931098
]
Hadoop QA commented on HDFS-14768:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 45s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 5 new + 172 unchanged - 0 fixed = 177 total (was 172) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 52s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestDFSStorageStateRecovery |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.namenode.TestRedudantBlocks |
| | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR |
| | hadoop.hdfs.server.namenode.TestUpgradeDomainBlockPlacementPolicy |
| | hadoop.hdfs.TestParallelUnixDomainRead |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.TestErasureCodingPolicies |
| | hadoop.hdfs.TestMiniDFSCluster |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestFileAppend2 |
| | hadoop.hdfs.TestDFSStripedOutputStream |
| | hadoop.hdfs.server.namenode.TestStartup |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy |
| | hadoop.hdfs.qjournal.client.TestQJMWithFaults |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:39e82acc485 |
| JIRA Issue | HDFS-14768 |
|
[jira] [Commented] (HDFS-14768) In some cases, erasure blocks are corruption when they are reconstruct.
[
https://issues.apache.org/jira/browse/HDFS-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931097#comment-16931097
]
Hadoop QA commented on HDFS-14768:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 5 new + 172 unchanged - 0 fixed = 177 total (was 172) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 83m
25s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:39e82acc485 |
| JIRA Issue | HDFS-14768 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12980458/HDFS-14768.0001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f6afc923239f 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2358e53 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/27891/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/27891/testReport/ |
| Max. process+thread count | 3421 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/27891/console |
| Powered by | Apache Yetus 0.8.0
[jira] [Work logged] (HDDS-2132) TestKeyValueContainer is failing
[
https://issues.apache.org/jira/browse/HDDS-2132?focusedWorklogId=313478=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313478
]
ASF GitHub Bot logged work on HDDS-2132:
Author: ASF GitHub Bot
Created on: 17/Sep/19 04:49
Start Date: 17/Sep/19 04:49
Worklog Time Spent: 10m
Work Description: adoroszlai commented on issue #1457: HDDS-2132.
TestKeyValueContainer is failing
URL: https://github.com/apache/hadoop/pull/1457#issuecomment-532057233
/label ozone
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313478)
Time Spent: 20m (was: 10m)
> TestKeyValueContainer is failing
>
>
> Key: HDDS-2132
> URL: https://issues.apache.org/jira/browse/HDDS-2132
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Doroszlai, Attila
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{TestKeyValueContainer}} is failing with the following exception
> {noformat}
> [ERROR]
> testContainerImportExport(org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer)
> Time elapsed: 0.173 s <<< ERROR!
> java.lang.NullPointerException
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
> at
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.parseKVContainerData(KeyValueContainerUtil.java:201)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.importContainerData(KeyValueContainer.java:500)
> at
> org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer.testContainerImportExport(TestKeyValueContainer.java:235)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2132) TestKeyValueContainer is failing
[
https://issues.apache.org/jira/browse/HDDS-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-2132:
-
Labels: pull-request-available (was: )
> TestKeyValueContainer is failing
>
>
> Key: HDDS-2132
> URL: https://issues.apache.org/jira/browse/HDDS-2132
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Priority: Major
> Labels: pull-request-available
>
> {{TestKeyValueContainer}} is failing with the following exception
> {noformat}
> [ERROR]
> testContainerImportExport(org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer)
> Time elapsed: 0.173 s <<< ERROR!
> java.lang.NullPointerException
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
> at
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.parseKVContainerData(KeyValueContainerUtil.java:201)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.importContainerData(KeyValueContainer.java:500)
> at
> org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer.testContainerImportExport(TestKeyValueContainer.java:235)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2132) TestKeyValueContainer is failing
[
https://issues.apache.org/jira/browse/HDDS-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doroszlai, Attila updated HDDS-2132:
Status: Patch Available (was: Open)
> TestKeyValueContainer is failing
>
>
> Key: HDDS-2132
> URL: https://issues.apache.org/jira/browse/HDDS-2132
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Doroszlai, Attila
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TestKeyValueContainer}} is failing with the following exception
> {noformat}
> [ERROR]
> testContainerImportExport(org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer)
> Time elapsed: 0.173 s <<< ERROR!
> java.lang.NullPointerException
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
> at
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.parseKVContainerData(KeyValueContainerUtil.java:201)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.importContainerData(KeyValueContainer.java:500)
> at
> org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer.testContainerImportExport(TestKeyValueContainer.java:235)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2132) TestKeyValueContainer is failing
[
https://issues.apache.org/jira/browse/HDDS-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doroszlai, Attila reassigned HDDS-2132:
---
Assignee: Doroszlai, Attila
> TestKeyValueContainer is failing
>
>
> Key: HDDS-2132
> URL: https://issues.apache.org/jira/browse/HDDS-2132
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Doroszlai, Attila
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TestKeyValueContainer}} is failing with the following exception
> {noformat}
> [ERROR]
> testContainerImportExport(org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer)
> Time elapsed: 0.173 s <<< ERROR!
> java.lang.NullPointerException
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
> at
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.parseKVContainerData(KeyValueContainerUtil.java:201)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.importContainerData(KeyValueContainer.java:500)
> at
> org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer.testContainerImportExport(TestKeyValueContainer.java:235)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2132) TestKeyValueContainer is failing
[
https://issues.apache.org/jira/browse/HDDS-2132?focusedWorklogId=313477=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313477
]
ASF GitHub Bot logged work on HDDS-2132:
Author: ASF GitHub Bot
Created on: 17/Sep/19 04:48
Start Date: 17/Sep/19 04:48
Worklog Time Spent: 10m
Work Description: adoroszlai commented on pull request #1457: HDDS-2132.
TestKeyValueContainer is failing
URL: https://github.com/apache/hadoop/pull/1457
## What changes were proposed in this pull request?
Fix unit tests recently broken by new `Preconditions.checkNotNull` in
`KeyValueContainerUtil#parseKVContainerData` (added in
fe8cdf0ab846df9c2f3f59d1d4875185633a27ea for
[HDDS-2076](https://issues.apache.org/jira/browse/HDDS-2076)).
https://issues.apache.org/jira/browse/HDDS-2132
https://issues.apache.org/jira/browse/HDDS-2133
## How was this patch tested?
```
$ mvn -am -Phdds -pl :hadoop-hdds-container-service clean test
...
[INFO] Apache Hadoop HDDS . SUCCESS [ 2.527
s]
[INFO] Apache Hadoop HDDS Config .. SUCCESS [ 3.223
s]
[INFO] Apache Hadoop HDDS Common .. SUCCESS [01:53
min]
[INFO] Apache Hadoop HDDS Server Framework SUCCESS [ 18.258
s]
[INFO] Apache Hadoop HDDS Container Service ... SUCCESS [01:13
min]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313477)
Remaining Estimate: 0h
Time Spent: 10m
> TestKeyValueContainer is failing
>
>
> Key: HDDS-2132
> URL: https://issues.apache.org/jira/browse/HDDS-2132
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TestKeyValueContainer}} is failing with the following exception
> {noformat}
> [ERROR]
> testContainerImportExport(org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer)
> Time elapsed: 0.173 s <<< ERROR!
> java.lang.NullPointerException
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
> at
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.parseKVContainerData(KeyValueContainerUtil.java:201)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.importContainerData(KeyValueContainer.java:500)
> at
> org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer.testContainerImportExport(TestKeyValueContainer.java:235)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
>
[jira] [Commented] (HDFS-6524) Choosing datanode retries times considering with block replica number
[
https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931084#comment-16931084
]
Hadoop QA commented on HDFS-6524:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
53s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 46s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.namenode.TestRedudantBlocks |
| |
hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
|
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR |
| | hadoop.hdfs.server.namenode.sps.TestStoragePolicySatisfierWithStripedFile
|
| | hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff |
| | hadoop.hdfs.server.namenode.TestFSImageWithSnapshot |
| | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion |
| |
[jira] [Commented] (HDFS-14814) RBF: The quota should be set the same as the nearest parent in Global Quota.
[
https://issues.apache.org/jira/browse/HDFS-14814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931078#comment-16931078
]
Jinglun commented on HDFS-14814:
Hi [~ayushtkn], thanks for your nice comments ! At present when we call
*Quota.setQuota(String path, long namespaceQuota, long storagespaceQuota,
StorageType type)*, all its children paths will be set too. So when a Router
Admin updates the quota of M1, the quota will be set on M1,M2,M3. Even there
isn't any quota in M3. If Router Admin updates quota of M1 and M2, then there
will be conflicts on M2's \{nquota} and M3's \{nquota,squota}. So I want the
inherited global quota to resolve the conflicts.
I agree we should let the Router Admin to explicitly define the quota. If we
have inherited global quota, he still can explicitly define quota by not using
QUOTA_UNSET. Then there won't be any inherit.
> RBF: The quota should be set the same as the nearest parent in Global Quota.
>
>
> Key: HDFS-14814
> URL: https://issues.apache.org/jira/browse/HDFS-14814
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HDFS-14814.001.patch, HDFS-14814.002.patch,
> HDFS-14814.003.patch
>
>
> I want to add a rule *'The quota should be set the same as the nearest
> parent'* to Global Quota. Supposing we have the mount table below.
> M1: /dir-a ns0->/dir-a \{nquota=10,squota=20}
> M2: /dir-a/dir-b ns1->/dir-b \{nquota=-1,squota=30}
> M3: /dir-a/dir-b/dir-c ns2->/dir-c \{nquota=-1,squota=-1}
> M4: /dir-d ns3->/dir-d \{nquota=-1,squota=-1}
>
> The quota for the remote locations on the namespaces should be:
> ns0->/dir-a \{nquota=10,squota=20}
> ns1->/dir-b \{nquota=10,squota=30}
> ns2->/dir-c \{nquota=10,squota=30}
> ns3->/dir-d \{nquota=-1,squota=-1}
>
> The quota of the remote location is set the same as the corresponding
> MountTable, and if there is no quota of the MountTable then the quota is set
> to the nearest parent MountTable with quota.
>
> It's easy to implement it. In RouterQuotaUpdateService each time we compute
> the currentQuotaUsage, we can get the quota info for each MountTable. We can
> do a
> check and fix all the MountTable which's quota doesn't match the rule above.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14814) RBF: The quota should be set the same as the nearest parent in Global Quota.
[
https://issues.apache.org/jira/browse/HDFS-14814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinglun updated HDFS-14814:
---
Description:
I want to add a rule *'The quota should be set the same as the nearest parent'*
to Global Quota. Supposing we have the mount table below.
M1: /dir-a ns0->/dir-a \{nquota=10,squota=20}
M2: /dir-a/dir-b ns1->/dir-b \{nquota=-1,squota=30}
M3: /dir-a/dir-b/dir-c ns2->/dir-c \{nquota=-1,squota=-1}
M4: /dir-d ns3->/dir-d \{nquota=-1,squota=-1}
The quota for the remote locations on the namespaces should be:
ns0->/dir-a \{nquota=10,squota=20}
ns1->/dir-b \{nquota=10,squota=30}
ns2->/dir-c \{nquota=10,squota=30}
ns3->/dir-d \{nquota=-1,squota=-1}
The quota of the remote location is set the same as the corresponding
MountTable, and if there is no quota of the MountTable then the quota is set to
the nearest parent MountTable with quota.
It's easy to implement it. In RouterQuotaUpdateService each time we compute the
currentQuotaUsage, we can get the quota info for each MountTable. We can do a
check and fix all the MountTable which's quota doesn't match the rule above.
was:
I want to add a rule *'The quota should be set the same as the nearest parent'*
to Global Quota. Supposing we have the mount table below.
/dir-a ns0->/dir-a \{nquota=10,squota=20}
/dir-a/dir-b ns1->/dir-b \{nquota=-1,squota=30}
/dir-a/dir-b/dir-c ns2->/dir-c \{nquota=-1,squota=-1}
/dir-d ns3->/dir-d \{nquota=-1,squota=-1}
The quota for the remote locations on the namespaces should be:
ns0->/dir-a \{nquota=10,squota=20}
ns1->/dir-b \{nquota=10,squota=30}
ns2->/dir-c \{nquota=10,squota=30}
ns3->/dir-d \{nquota=-1,squota=-1}
The quota of the remote location is set the same as the corresponding
MountTable, and if there is no quota of the MountTable then the quota is set to
the nearest parent MountTable with quota.
It's easy to implement it. In RouterQuotaUpdateService each time we compute the
currentQuotaUsage, we can get the quota info for each MountTable. We can do a
check and fix all the MountTable which's quota doesn't match the rule above.
> RBF: The quota should be set the same as the nearest parent in Global Quota.
>
>
> Key: HDFS-14814
> URL: https://issues.apache.org/jira/browse/HDFS-14814
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HDFS-14814.001.patch, HDFS-14814.002.patch,
> HDFS-14814.003.patch
>
>
> I want to add a rule *'The quota should be set the same as the nearest
> parent'* to Global Quota. Supposing we have the mount table below.
> M1: /dir-a ns0->/dir-a \{nquota=10,squota=20}
> M2: /dir-a/dir-b ns1->/dir-b \{nquota=-1,squota=30}
> M3: /dir-a/dir-b/dir-c ns2->/dir-c \{nquota=-1,squota=-1}
> M4: /dir-d ns3->/dir-d \{nquota=-1,squota=-1}
>
> The quota for the remote locations on the namespaces should be:
> ns0->/dir-a \{nquota=10,squota=20}
> ns1->/dir-b \{nquota=10,squota=30}
> ns2->/dir-c \{nquota=10,squota=30}
> ns3->/dir-d \{nquota=-1,squota=-1}
>
> The quota of the remote location is set the same as the corresponding
> MountTable, and if there is no quota of the MountTable then the quota is set
> to the nearest parent MountTable with quota.
>
> It's easy to implement it. In RouterQuotaUpdateService each time we compute
> the currentQuotaUsage, we can get the quota info for each MountTable. We can
> do a
> check and fix all the MountTable which's quota doesn't match the rule above.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14818) Check native pmdk lib by 'hadoop checknative' command
[
https://issues.apache.org/jira/browse/HDFS-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931065#comment-16931065
]
Hadoop QA commented on HDFS-14818:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 21m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 21m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 21s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
58s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}119m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ipc.TestIPC |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:39e82acc485 |
| JIRA Issue | HDFS-14818 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12980454/HDFS-14818.001.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
javadoc mvninstall shadedclient findbugs checkstyle |
| uname | Linux 6a1b1e85b1f4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2358e53 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/27888/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/27888/testReport/ |
| Max. process+thread count | 1343 (vs. ulimit of 5500) |
| modules
[jira] [Commented] (HDFS-14813) RBF: Make Global quota and Remote quota consistent.
[ https://issues.apache.org/jira/browse/HDFS-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931063#comment-16931063 ] Jinglun commented on HDFS-14813: My bad, I miss [~ayushtkn] comments in HDFS-14814. Let's discuss inherit quota in HDFS-14814, I'll move my comments there. > RBF: Make Global quota and Remote quota consistent. > --- > > Key: HDFS-14813 > URL: https://issues.apache.org/jira/browse/HDFS-14813 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > > Make Global quota and remote quota consistent. > (Global quota: the quota on mount table, Remote quota: the quota on > namespace) > HDFS administrator can use global quota to simplify the management for > federation paths. But there is no consistent constraint for the global quota > and the remote quota. As an hdfs administrator the inconsistent has 3 > disadvantages on management: > 1. The quota part of getQuotaUsage() on a federation path is not > helpful. It's neither the global quota nor one of the remote quotas. > 2. The global quota could be different with the remote quota. When a > QuotaExceedException happens it needs the administrator to find out whether > it's a violation of the global quota or the remote quota. > 3. For management simplicity, it's always a good idea to keep the global > quota and the remote quota the same. Now we need the administrator to keep > the consistent manually. > My proposal is to add a constraint for global quota: > 1. For federation paths, global quota could be inherited from parent > federation path. > 2. For all remote paths in mount tables, the remote quotas must be > consistent with the global quotas. > To implement this, my idea is: > 1. Global quota could be inherited. Add a method getGlobalQuota(String > path) to Quota.java returning the global quota. > 2. Each time RouterQuotaUpdateService updates the quota usage for mount > table entries, it also checks and updates the remote quota. > 3. When getQuotaUsage() on a federation path, return the global quota. > 4. When setQuota() on a federation path, first update the global quota > in mount table, then recompute global quota for the current path and its > children paths, finally update all the federation paths. > > Implement 1+2 in HDFS-14814 > Implement 4 in HDFS-14815 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14568) setStoragePolicy should check quota and update consume on storage type quota.
[
https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931056#comment-16931056
]
Jinglun commented on HDFS-14568:
Hi [~xyao] [~surendrasingh], do you have time for this? Could you help to
review v04? Thanks!
> setStoragePolicy should check quota and update consume on storage type quota.
> -
>
> Key: HDFS-14568
> URL: https://issues.apache.org/jira/browse/HDFS-14568
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Labels: imcompatible
> Attachments: HDFS-14568-001.patch, HDFS-14568-unit-test.patch,
> HDFS-14568.002.patch, HDFS-14568.003.patch, HDFS-14568.004.patch
>
>
> The quota and consume of the file's ancestors are not handled when the
> storage policy of the file is changed. For example:
> 1. Set quota StorageType.SSD fileSpace-1 to the parent dir;
> 2. Create a file size of fileSpace with storage policy \{DISK,DISK,DISK}
> under it;
> 3. Change the storage policy of the file to ALLSSD_STORAGE_POLICY_NAME and
> expect a QuotaByStorageTypeExceededException.
> Because the quota and consume is not handled, the expected exception is not
> threw out.
>
> There are 3 reasons why we should handle the consume and the quota.
> 1. Replication uses the new storage policy. Considering a file with BlockType
> CONTIGUOUS. It's replication factor is 3 and it's storage policy is "HOT".
> Now we change the policy to "ONE_SSD". If a DN goes down and the file needs
> replication, the NN will choose storages in policy "ONE_SSD" and replicate
> the block to a SSD storage.
> 2. We acturally have a cluster storaging both HOT and COLD data. We have a
> backgroud process searching all the files to find those that are not accessed
> for a period of time. Then we set them to COLD and start a mover to move the
> replicas. After moving, all the replicas are consistent with the storage
> policy.
> 3. The NameNode manages the global state of the cluster. If there is any
> inconsistent situation, such as the replicas doesn't match the storage policy
> of the file, we should take the NameNode as the standard and make the cluster
> to match the NameNode. The block replication is a good example of the rule.
> When we count the consume of a file(CONTIGUOUS), we multiply the replication
> factor with the file's length, no matter the file is under replicated or
> excessed. So does the storage type quota and consume.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14568) setStoragePolicy should check quota and update consume on storage type quota.
[
https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinglun updated HDFS-14568:
---
Labels: imcompatible (was: )
> setStoragePolicy should check quota and update consume on storage type quota.
> -
>
> Key: HDFS-14568
> URL: https://issues.apache.org/jira/browse/HDFS-14568
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Labels: imcompatible
> Attachments: HDFS-14568-001.patch, HDFS-14568-unit-test.patch,
> HDFS-14568.002.patch, HDFS-14568.003.patch, HDFS-14568.004.patch
>
>
> The quota and consume of the file's ancestors are not handled when the
> storage policy of the file is changed. For example:
> 1. Set quota StorageType.SSD fileSpace-1 to the parent dir;
> 2. Create a file size of fileSpace with storage policy \{DISK,DISK,DISK}
> under it;
> 3. Change the storage policy of the file to ALLSSD_STORAGE_POLICY_NAME and
> expect a QuotaByStorageTypeExceededException.
> Because the quota and consume is not handled, the expected exception is not
> threw out.
>
> There are 3 reasons why we should handle the consume and the quota.
> 1. Replication uses the new storage policy. Considering a file with BlockType
> CONTIGUOUS. It's replication factor is 3 and it's storage policy is "HOT".
> Now we change the policy to "ONE_SSD". If a DN goes down and the file needs
> replication, the NN will choose storages in policy "ONE_SSD" and replicate
> the block to a SSD storage.
> 2. We acturally have a cluster storaging both HOT and COLD data. We have a
> backgroud process searching all the files to find those that are not accessed
> for a period of time. Then we set them to COLD and start a mover to move the
> replicas. After moving, all the replicas are consistent with the storage
> policy.
> 3. The NameNode manages the global state of the cluster. If there is any
> inconsistent situation, such as the replicas doesn't match the storage policy
> of the file, we should take the NameNode as the standard and make the cluster
> to match the NameNode. The block replication is a good example of the rule.
> When we count the consume of a file(CONTIGUOUS), we multiply the replication
> factor with the file's length, no matter the file is under replicated or
> excessed. So does the storage type quota and consume.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14813) RBF: Make Global quota and Remote quota consistent.
[
https://issues.apache.org/jira/browse/HDFS-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931054#comment-16931054
]
Jinglun commented on HDFS-14813:
Hi [~ayushtkn], very sorry for my late response. I want to do 1+2 together
because I need 1 to resolve global quota conflicts. For example I have mount
tables below, the question is what quotas do I expect for ns1->/dir-b and
ns2->/dir-c.
{quote}M1. /dir-a ns0->/dir-a \{nquota=10,squota=20}
M2. /dir-a/dir-b ns1->/dir-b \{nquota=-1, squota=30}
M3. /dir-a/dir-b/dir-c ns2->/dir-c \{nquota=-1, squota=-1 }
{quote}
At present when we call Quota.setQuota(), all its children paths will be set
too. Quota in M1 will be set on M1,M2,M3 and quota in M2 will be set on M2,M3.
So there will be conflicts on M2's \{nquota} and M3's \{nquota,squota}. I want
to use inherited global quota to resolve this. If one mount table's global
quota is unset, it inherits its parent mount table's global quota.
> RBF: Make Global quota and Remote quota consistent.
> ---
>
> Key: HDFS-14813
> URL: https://issues.apache.org/jira/browse/HDFS-14813
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
>
> Make Global quota and remote quota consistent.
> (Global quota: the quota on mount table, Remote quota: the quota on
> namespace)
> HDFS administrator can use global quota to simplify the management for
> federation paths. But there is no consistent constraint for the global quota
> and the remote quota. As an hdfs administrator the inconsistent has 3
> disadvantages on management:
> 1. The quota part of getQuotaUsage() on a federation path is not
> helpful. It's neither the global quota nor one of the remote quotas.
> 2. The global quota could be different with the remote quota. When a
> QuotaExceedException happens it needs the administrator to find out whether
> it's a violation of the global quota or the remote quota.
> 3. For management simplicity, it's always a good idea to keep the global
> quota and the remote quota the same. Now we need the administrator to keep
> the consistent manually.
> My proposal is to add a constraint for global quota:
> 1. For federation paths, global quota could be inherited from parent
> federation path.
> 2. For all remote paths in mount tables, the remote quotas must be
> consistent with the global quotas.
> To implement this, my idea is:
> 1. Global quota could be inherited. Add a method getGlobalQuota(String
> path) to Quota.java returning the global quota.
> 2. Each time RouterQuotaUpdateService updates the quota usage for mount
> table entries, it also checks and updates the remote quota.
> 3. When getQuotaUsage() on a federation path, return the global quota.
> 4. When setQuota() on a federation path, first update the global quota
> in mount table, then recompute global quota for the current path and its
> children paths, finally update all the federation paths.
>
> Implement 1+2 in HDFS-14814
> Implement 4 in HDFS-14815
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2020) Remove mTLS from Ozone GRPC
[ https://issues.apache.org/jira/browse/HDDS-2020?focusedWorklogId=313458=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313458 ] ASF GitHub Bot logged work on HDDS-2020: Author: ASF GitHub Bot Created on: 17/Sep/19 03:33 Start Date: 17/Sep/19 03:33 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1369: HDDS-2020. Remove mTLS from Ozone GRPC. Contributed by Xiaoyu Yao. URL: https://github.com/apache/hadoop/pull/1369#issuecomment-532044005 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 2458 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 2 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 12 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 73 | Maven dependency ordering for branch | | -1 | mvninstall | 30 | hadoop-ozone in trunk failed. | | -1 | compile | 23 | hadoop-ozone in trunk failed. | | -0 | checkstyle | 59 | The patch fails to run checkstyle in hadoop-ozone | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 887 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 84 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 164 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 27 | hadoop-ozone in trunk failed. | | -0 | patch | 197 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for patch | | -1 | mvninstall | 33 | hadoop-ozone in the patch failed. | | -1 | compile | 26 | hadoop-ozone in the patch failed. | | -1 | cc | 26 | hadoop-ozone in the patch failed. | | -1 | javac | 26 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 50 | hadoop-hdds: The patch generated 62 new + 919 unchanged - 44 fixed = 981 total (was 963) | | -0 | checkstyle | 51 | The patch fails to run checkstyle in hadoop-ozone | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 702 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 84 | hadoop-ozone in the patch failed. | | -1 | findbugs | 169 | hadoop-hdds generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | -1 | findbugs | 27 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 166 | hadoop-hdds in the patch failed. | | -1 | unit | 29 | hadoop-ozone in the patch failed. | | +1 | asflicense | 35 | The patch does not generate ASF License warnings. | | | | 5747 | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-hdds | | | Boxing/unboxing to parse a primitive org.apache.hadoop.hdds.security.x509.certificate.client.DefaultCertificateClient.loadAllCertificates() At DefaultCertificateClient.java:org.apache.hadoop.hdds.security.x509.certificate.client.DefaultCertificateClient.loadAllCertificates() At DefaultCertificateClient.java:[line 141] | | Failed junit tests | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.container.keyvalue.TestKeyValueContainer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1369/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1369 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc | | uname | Linux a3af19ce03e3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 2358e53 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1369/8/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1369/8/artifact/out/branch-compile-hadoop-ozone.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1369/8/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1369/out/maven-branch-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1369/8/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs |
[jira] [Comment Edited] (HDFS-14768) In some cases, erasure blocks are corruption when they are reconstruct.
[
https://issues.apache.org/jira/browse/HDFS-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931021#comment-16931021
]
guojh edited comment on HDFS-14768 at 9/17/19 2:34 AM:
---
[~surendrasingh] I put a new patch based on the latest code. Could you help to
review it?
was (Author: gjhkael):
[~surendrasingh] I put a new patch based on the latest code.
> In some cases, erasure blocks are corruption when they are reconstruct.
>
>
> Key: HDFS-14768
> URL: https://issues.apache.org/jira/browse/HDFS-14768
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding, hdfs, namenode
>Affects Versions: 3.0.2
>Reporter: guojh
>Assignee: guojh
>Priority: Major
> Labels: patch
> Fix For: 3.3.0
>
> Attachments: 1568275810244.jpg, 1568276338275.jpg,
> HDFS-14768.000.patch, HDFS-14768.0001.patch, guojh_UT_after_deomission.txt,
> guojh_UT_before_deomission.txt, zhaoyiming_UT_after_deomission.txt,
> zhaoyiming_UT_beofre_deomission.txt
>
>
> Policy is RS-6-3-1024K, version is hadoop 3.0.2;
> We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
> index[3,4], increase the index 6 datanode's
> pendingReplicationWithoutTargets that make it large than
> replicationStreamsHardLimit(we set 14). Then, After the method
> chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
> [0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
> In method scheduleReconstruction of BlockManager, the additionalReplRequired
> is 9 - 7 = 2. After Namenode choose two target Datanode, will assign a
> erasureCode task to target datanode.
> When datanode get the task will build targetIndices from liveBlockIndices
> and target length. the code is blow.
> {code:java}
> // code placeholder
> targetIndices = new short[targets.length];
> private void initTargetIndices() {
> BitSet bitset = reconstructor.getLiveBitSet();
> int m = 0; hasValidTargets = false;
> for (int i = 0; i < dataBlkNum + parityBlkNum; i++) {
> if (!bitset.get) {
> if (reconstructor.getBlockLen > 0) {
> if (m < targets.length) {
> targetIndices[m++] = (short)i;
> hasValidTargets = true;
> }
> }
> }
> }
> {code}
> targetIndices[0]=6, and targetIndices[1] is aways 0 from initial value.
> The StripedReader is aways create reader from first 6 index block, and is
> [0,1,2,3,4,5]
> Use the index [0,1,2,3,4,5] to build target index[6,0] will trigger the isal
> bug. the block index6's data is corruption(all data is zero).
> I write a unit test can stabilize repreduce.
> {code:java}
> // code placeholder
> private int replicationStreamsHardLimit =
> DFSConfigKeys.DFS_NAMENODE_REPLICATION_STREAMS_HARD_LIMIT_DEFAULT;
> numDNs = dataBlocks + parityBlocks + 10;
> @Test(timeout = 24)
> public void testFileDecommission() throws Exception {
> LOG.info("Starting test testFileDecommission");
> final Path ecFile = new Path(ecDir, "testFileDecommission");
> int writeBytes = cellSize * dataBlocks;
> writeStripedFile(dfs, ecFile, writeBytes);
> Assert.assertEquals(0, bm.numOfUnderReplicatedBlocks());
> FileChecksum fileChecksum1 = dfs.getFileChecksum(ecFile, writeBytes);
> final INodeFile fileNode = cluster.getNamesystem().getFSDirectory()
> .getINode4Write(ecFile.toString()).asFile();
> LocatedBlocks locatedBlocks =
> StripedFileTestUtil.getLocatedBlocks(ecFile, dfs);
> LocatedBlock lb = dfs.getClient().getLocatedBlocks(ecFile.toString(), 0)
> .get(0);
> DatanodeInfo[] dnLocs = lb.getLocations();
> LocatedStripedBlock lastBlock =
> (LocatedStripedBlock)locatedBlocks.getLastLocatedBlock();
> DatanodeInfo[] storageInfos = lastBlock.getLocations();
> //
> DatanodeDescriptor datanodeDescriptor =
> cluster.getNameNode().getNamesystem()
>
> .getBlockManager().getDatanodeManager().getDatanode(storageInfos[6].getDatanodeUuid());
> BlockInfo firstBlock = fileNode.getBlocks()[0];
> DatanodeStorageInfo[] dStorageInfos = bm.getStorages(firstBlock);
> // the first heartbeat will consume 3 replica tasks
> for (int i = 0; i <= replicationStreamsHardLimit + 3; i++) {
> BlockManagerTestUtil.addBlockToBeReplicated(datanodeDescriptor, new
> Block(i),
> new DatanodeStorageInfo[]{dStorageInfos[0]});
> }
> assertEquals(dataBlocks + parityBlocks, dnLocs.length);
> int[] decommNodeIndex = {3, 4};
> final List decommisionNodes = new ArrayList();
> // add the node which will be decommissioning
> decommisionNodes.add(dnLocs[decommNodeIndex[0]]);
> decommisionNodes.add(dnLocs[decommNodeIndex[1]]);
> decommissionNode(0, decommisionNodes, AdminStates.DECOMMISSIONED);
>
[jira] [Commented] (HDFS-14768) In some cases, erasure blocks are corruption when they are reconstruct.
[
https://issues.apache.org/jira/browse/HDFS-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931021#comment-16931021
]
guojh commented on HDFS-14768:
--
[~surendrasingh] I put a new patch based on the latest code.
> In some cases, erasure blocks are corruption when they are reconstruct.
>
>
> Key: HDFS-14768
> URL: https://issues.apache.org/jira/browse/HDFS-14768
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding, hdfs, namenode
>Affects Versions: 3.0.2
>Reporter: guojh
>Assignee: guojh
>Priority: Major
> Labels: patch
> Fix For: 3.3.0
>
> Attachments: 1568275810244.jpg, 1568276338275.jpg,
> HDFS-14768.000.patch, HDFS-14768.0001.patch, guojh_UT_after_deomission.txt,
> guojh_UT_before_deomission.txt, zhaoyiming_UT_after_deomission.txt,
> zhaoyiming_UT_beofre_deomission.txt
>
>
> Policy is RS-6-3-1024K, version is hadoop 3.0.2;
> We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
> index[3,4], increase the index 6 datanode's
> pendingReplicationWithoutTargets that make it large than
> replicationStreamsHardLimit(we set 14). Then, After the method
> chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
> [0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
> In method scheduleReconstruction of BlockManager, the additionalReplRequired
> is 9 - 7 = 2. After Namenode choose two target Datanode, will assign a
> erasureCode task to target datanode.
> When datanode get the task will build targetIndices from liveBlockIndices
> and target length. the code is blow.
> {code:java}
> // code placeholder
> targetIndices = new short[targets.length];
> private void initTargetIndices() {
> BitSet bitset = reconstructor.getLiveBitSet();
> int m = 0; hasValidTargets = false;
> for (int i = 0; i < dataBlkNum + parityBlkNum; i++) {
> if (!bitset.get) {
> if (reconstructor.getBlockLen > 0) {
> if (m < targets.length) {
> targetIndices[m++] = (short)i;
> hasValidTargets = true;
> }
> }
> }
> }
> {code}
> targetIndices[0]=6, and targetIndices[1] is aways 0 from initial value.
> The StripedReader is aways create reader from first 6 index block, and is
> [0,1,2,3,4,5]
> Use the index [0,1,2,3,4,5] to build target index[6,0] will trigger the isal
> bug. the block index6's data is corruption(all data is zero).
> I write a unit test can stabilize repreduce.
> {code:java}
> // code placeholder
> private int replicationStreamsHardLimit =
> DFSConfigKeys.DFS_NAMENODE_REPLICATION_STREAMS_HARD_LIMIT_DEFAULT;
> numDNs = dataBlocks + parityBlocks + 10;
> @Test(timeout = 24)
> public void testFileDecommission() throws Exception {
> LOG.info("Starting test testFileDecommission");
> final Path ecFile = new Path(ecDir, "testFileDecommission");
> int writeBytes = cellSize * dataBlocks;
> writeStripedFile(dfs, ecFile, writeBytes);
> Assert.assertEquals(0, bm.numOfUnderReplicatedBlocks());
> FileChecksum fileChecksum1 = dfs.getFileChecksum(ecFile, writeBytes);
> final INodeFile fileNode = cluster.getNamesystem().getFSDirectory()
> .getINode4Write(ecFile.toString()).asFile();
> LocatedBlocks locatedBlocks =
> StripedFileTestUtil.getLocatedBlocks(ecFile, dfs);
> LocatedBlock lb = dfs.getClient().getLocatedBlocks(ecFile.toString(), 0)
> .get(0);
> DatanodeInfo[] dnLocs = lb.getLocations();
> LocatedStripedBlock lastBlock =
> (LocatedStripedBlock)locatedBlocks.getLastLocatedBlock();
> DatanodeInfo[] storageInfos = lastBlock.getLocations();
> //
> DatanodeDescriptor datanodeDescriptor =
> cluster.getNameNode().getNamesystem()
>
> .getBlockManager().getDatanodeManager().getDatanode(storageInfos[6].getDatanodeUuid());
> BlockInfo firstBlock = fileNode.getBlocks()[0];
> DatanodeStorageInfo[] dStorageInfos = bm.getStorages(firstBlock);
> // the first heartbeat will consume 3 replica tasks
> for (int i = 0; i <= replicationStreamsHardLimit + 3; i++) {
> BlockManagerTestUtil.addBlockToBeReplicated(datanodeDescriptor, new
> Block(i),
> new DatanodeStorageInfo[]{dStorageInfos[0]});
> }
> assertEquals(dataBlocks + parityBlocks, dnLocs.length);
> int[] decommNodeIndex = {3, 4};
> final List decommisionNodes = new ArrayList();
> // add the node which will be decommissioning
> decommisionNodes.add(dnLocs[decommNodeIndex[0]]);
> decommisionNodes.add(dnLocs[decommNodeIndex[1]]);
> decommissionNode(0, decommisionNodes, AdminStates.DECOMMISSIONED);
> assertEquals(decommisionNodes.size(), fsn.getNumDecomLiveDataNodes());
> bm.getDatanodeManager().removeDatanode(datanodeDescriptor);
> //assertNull(checkFile(dfs, ecFile, 9, decommisionNodes,
[jira] [Updated] (HDFS-14768) In some cases, erasure blocks are corruption when they are reconstruct.
[
https://issues.apache.org/jira/browse/HDFS-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
guojh updated HDFS-14768:
-
Attachment: (was: HDFS-14768.0001.patch)
> In some cases, erasure blocks are corruption when they are reconstruct.
>
>
> Key: HDFS-14768
> URL: https://issues.apache.org/jira/browse/HDFS-14768
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding, hdfs, namenode
>Affects Versions: 3.0.2
>Reporter: guojh
>Assignee: guojh
>Priority: Major
> Labels: patch
> Fix For: 3.3.0
>
> Attachments: 1568275810244.jpg, 1568276338275.jpg,
> HDFS-14768.000.patch, HDFS-14768.0001.patch, guojh_UT_after_deomission.txt,
> guojh_UT_before_deomission.txt, zhaoyiming_UT_after_deomission.txt,
> zhaoyiming_UT_beofre_deomission.txt
>
>
> Policy is RS-6-3-1024K, version is hadoop 3.0.2;
> We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
> index[3,4], increase the index 6 datanode's
> pendingReplicationWithoutTargets that make it large than
> replicationStreamsHardLimit(we set 14). Then, After the method
> chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
> [0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
> In method scheduleReconstruction of BlockManager, the additionalReplRequired
> is 9 - 7 = 2. After Namenode choose two target Datanode, will assign a
> erasureCode task to target datanode.
> When datanode get the task will build targetIndices from liveBlockIndices
> and target length. the code is blow.
> {code:java}
> // code placeholder
> targetIndices = new short[targets.length];
> private void initTargetIndices() {
> BitSet bitset = reconstructor.getLiveBitSet();
> int m = 0; hasValidTargets = false;
> for (int i = 0; i < dataBlkNum + parityBlkNum; i++) {
> if (!bitset.get) {
> if (reconstructor.getBlockLen > 0) {
> if (m < targets.length) {
> targetIndices[m++] = (short)i;
> hasValidTargets = true;
> }
> }
> }
> }
> {code}
> targetIndices[0]=6, and targetIndices[1] is aways 0 from initial value.
> The StripedReader is aways create reader from first 6 index block, and is
> [0,1,2,3,4,5]
> Use the index [0,1,2,3,4,5] to build target index[6,0] will trigger the isal
> bug. the block index6's data is corruption(all data is zero).
> I write a unit test can stabilize repreduce.
> {code:java}
> // code placeholder
> private int replicationStreamsHardLimit =
> DFSConfigKeys.DFS_NAMENODE_REPLICATION_STREAMS_HARD_LIMIT_DEFAULT;
> numDNs = dataBlocks + parityBlocks + 10;
> @Test(timeout = 24)
> public void testFileDecommission() throws Exception {
> LOG.info("Starting test testFileDecommission");
> final Path ecFile = new Path(ecDir, "testFileDecommission");
> int writeBytes = cellSize * dataBlocks;
> writeStripedFile(dfs, ecFile, writeBytes);
> Assert.assertEquals(0, bm.numOfUnderReplicatedBlocks());
> FileChecksum fileChecksum1 = dfs.getFileChecksum(ecFile, writeBytes);
> final INodeFile fileNode = cluster.getNamesystem().getFSDirectory()
> .getINode4Write(ecFile.toString()).asFile();
> LocatedBlocks locatedBlocks =
> StripedFileTestUtil.getLocatedBlocks(ecFile, dfs);
> LocatedBlock lb = dfs.getClient().getLocatedBlocks(ecFile.toString(), 0)
> .get(0);
> DatanodeInfo[] dnLocs = lb.getLocations();
> LocatedStripedBlock lastBlock =
> (LocatedStripedBlock)locatedBlocks.getLastLocatedBlock();
> DatanodeInfo[] storageInfos = lastBlock.getLocations();
> //
> DatanodeDescriptor datanodeDescriptor =
> cluster.getNameNode().getNamesystem()
>
> .getBlockManager().getDatanodeManager().getDatanode(storageInfos[6].getDatanodeUuid());
> BlockInfo firstBlock = fileNode.getBlocks()[0];
> DatanodeStorageInfo[] dStorageInfos = bm.getStorages(firstBlock);
> // the first heartbeat will consume 3 replica tasks
> for (int i = 0; i <= replicationStreamsHardLimit + 3; i++) {
> BlockManagerTestUtil.addBlockToBeReplicated(datanodeDescriptor, new
> Block(i),
> new DatanodeStorageInfo[]{dStorageInfos[0]});
> }
> assertEquals(dataBlocks + parityBlocks, dnLocs.length);
> int[] decommNodeIndex = {3, 4};
> final List decommisionNodes = new ArrayList();
> // add the node which will be decommissioning
> decommisionNodes.add(dnLocs[decommNodeIndex[0]]);
> decommisionNodes.add(dnLocs[decommNodeIndex[1]]);
> decommissionNode(0, decommisionNodes, AdminStates.DECOMMISSIONED);
> assertEquals(decommisionNodes.size(), fsn.getNumDecomLiveDataNodes());
> bm.getDatanodeManager().removeDatanode(datanodeDescriptor);
> //assertNull(checkFile(dfs, ecFile, 9, decommisionNodes, numDNs));
> // Ensure decommissioned datanode is not
[jira] [Updated] (HDFS-14768) In some cases, erasure blocks are corruption when they are reconstruct.
[
https://issues.apache.org/jira/browse/HDFS-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
guojh updated HDFS-14768:
-
Attachment: HDFS-14768.0001.patch
> In some cases, erasure blocks are corruption when they are reconstruct.
>
>
> Key: HDFS-14768
> URL: https://issues.apache.org/jira/browse/HDFS-14768
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding, hdfs, namenode
>Affects Versions: 3.0.2
>Reporter: guojh
>Assignee: guojh
>Priority: Major
> Labels: patch
> Fix For: 3.3.0
>
> Attachments: 1568275810244.jpg, 1568276338275.jpg,
> HDFS-14768.000.patch, HDFS-14768.0001.patch, guojh_UT_after_deomission.txt,
> guojh_UT_before_deomission.txt, zhaoyiming_UT_after_deomission.txt,
> zhaoyiming_UT_beofre_deomission.txt
>
>
> Policy is RS-6-3-1024K, version is hadoop 3.0.2;
> We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
> index[3,4], increase the index 6 datanode's
> pendingReplicationWithoutTargets that make it large than
> replicationStreamsHardLimit(we set 14). Then, After the method
> chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
> [0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
> In method scheduleReconstruction of BlockManager, the additionalReplRequired
> is 9 - 7 = 2. After Namenode choose two target Datanode, will assign a
> erasureCode task to target datanode.
> When datanode get the task will build targetIndices from liveBlockIndices
> and target length. the code is blow.
> {code:java}
> // code placeholder
> targetIndices = new short[targets.length];
> private void initTargetIndices() {
> BitSet bitset = reconstructor.getLiveBitSet();
> int m = 0; hasValidTargets = false;
> for (int i = 0; i < dataBlkNum + parityBlkNum; i++) {
> if (!bitset.get) {
> if (reconstructor.getBlockLen > 0) {
> if (m < targets.length) {
> targetIndices[m++] = (short)i;
> hasValidTargets = true;
> }
> }
> }
> }
> {code}
> targetIndices[0]=6, and targetIndices[1] is aways 0 from initial value.
> The StripedReader is aways create reader from first 6 index block, and is
> [0,1,2,3,4,5]
> Use the index [0,1,2,3,4,5] to build target index[6,0] will trigger the isal
> bug. the block index6's data is corruption(all data is zero).
> I write a unit test can stabilize repreduce.
> {code:java}
> // code placeholder
> private int replicationStreamsHardLimit =
> DFSConfigKeys.DFS_NAMENODE_REPLICATION_STREAMS_HARD_LIMIT_DEFAULT;
> numDNs = dataBlocks + parityBlocks + 10;
> @Test(timeout = 24)
> public void testFileDecommission() throws Exception {
> LOG.info("Starting test testFileDecommission");
> final Path ecFile = new Path(ecDir, "testFileDecommission");
> int writeBytes = cellSize * dataBlocks;
> writeStripedFile(dfs, ecFile, writeBytes);
> Assert.assertEquals(0, bm.numOfUnderReplicatedBlocks());
> FileChecksum fileChecksum1 = dfs.getFileChecksum(ecFile, writeBytes);
> final INodeFile fileNode = cluster.getNamesystem().getFSDirectory()
> .getINode4Write(ecFile.toString()).asFile();
> LocatedBlocks locatedBlocks =
> StripedFileTestUtil.getLocatedBlocks(ecFile, dfs);
> LocatedBlock lb = dfs.getClient().getLocatedBlocks(ecFile.toString(), 0)
> .get(0);
> DatanodeInfo[] dnLocs = lb.getLocations();
> LocatedStripedBlock lastBlock =
> (LocatedStripedBlock)locatedBlocks.getLastLocatedBlock();
> DatanodeInfo[] storageInfos = lastBlock.getLocations();
> //
> DatanodeDescriptor datanodeDescriptor =
> cluster.getNameNode().getNamesystem()
>
> .getBlockManager().getDatanodeManager().getDatanode(storageInfos[6].getDatanodeUuid());
> BlockInfo firstBlock = fileNode.getBlocks()[0];
> DatanodeStorageInfo[] dStorageInfos = bm.getStorages(firstBlock);
> // the first heartbeat will consume 3 replica tasks
> for (int i = 0; i <= replicationStreamsHardLimit + 3; i++) {
> BlockManagerTestUtil.addBlockToBeReplicated(datanodeDescriptor, new
> Block(i),
> new DatanodeStorageInfo[]{dStorageInfos[0]});
> }
> assertEquals(dataBlocks + parityBlocks, dnLocs.length);
> int[] decommNodeIndex = {3, 4};
> final List decommisionNodes = new ArrayList();
> // add the node which will be decommissioning
> decommisionNodes.add(dnLocs[decommNodeIndex[0]]);
> decommisionNodes.add(dnLocs[decommNodeIndex[1]]);
> decommissionNode(0, decommisionNodes, AdminStates.DECOMMISSIONED);
> assertEquals(decommisionNodes.size(), fsn.getNumDecomLiveDataNodes());
> bm.getDatanodeManager().removeDatanode(datanodeDescriptor);
> //assertNull(checkFile(dfs, ecFile, 9, decommisionNodes, numDNs));
> // Ensure decommissioned datanode is not
[jira] [Updated] (HDFS-14768) In some cases, erasure blocks are corruption when they are reconstruct.
[
https://issues.apache.org/jira/browse/HDFS-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
guojh updated HDFS-14768:
-
Attachment: HDFS-14768.0001.patch
> In some cases, erasure blocks are corruption when they are reconstruct.
>
>
> Key: HDFS-14768
> URL: https://issues.apache.org/jira/browse/HDFS-14768
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding, hdfs, namenode
>Affects Versions: 3.0.2
>Reporter: guojh
>Assignee: guojh
>Priority: Major
> Labels: patch
> Fix For: 3.3.0
>
> Attachments: 1568275810244.jpg, 1568276338275.jpg,
> HDFS-14768.000.patch, HDFS-14768.0001.patch, guojh_UT_after_deomission.txt,
> guojh_UT_before_deomission.txt, zhaoyiming_UT_after_deomission.txt,
> zhaoyiming_UT_beofre_deomission.txt
>
>
> Policy is RS-6-3-1024K, version is hadoop 3.0.2;
> We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
> index[3,4], increase the index 6 datanode's
> pendingReplicationWithoutTargets that make it large than
> replicationStreamsHardLimit(we set 14). Then, After the method
> chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
> [0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
> In method scheduleReconstruction of BlockManager, the additionalReplRequired
> is 9 - 7 = 2. After Namenode choose two target Datanode, will assign a
> erasureCode task to target datanode.
> When datanode get the task will build targetIndices from liveBlockIndices
> and target length. the code is blow.
> {code:java}
> // code placeholder
> targetIndices = new short[targets.length];
> private void initTargetIndices() {
> BitSet bitset = reconstructor.getLiveBitSet();
> int m = 0; hasValidTargets = false;
> for (int i = 0; i < dataBlkNum + parityBlkNum; i++) {
> if (!bitset.get) {
> if (reconstructor.getBlockLen > 0) {
> if (m < targets.length) {
> targetIndices[m++] = (short)i;
> hasValidTargets = true;
> }
> }
> }
> }
> {code}
> targetIndices[0]=6, and targetIndices[1] is aways 0 from initial value.
> The StripedReader is aways create reader from first 6 index block, and is
> [0,1,2,3,4,5]
> Use the index [0,1,2,3,4,5] to build target index[6,0] will trigger the isal
> bug. the block index6's data is corruption(all data is zero).
> I write a unit test can stabilize repreduce.
> {code:java}
> // code placeholder
> private int replicationStreamsHardLimit =
> DFSConfigKeys.DFS_NAMENODE_REPLICATION_STREAMS_HARD_LIMIT_DEFAULT;
> numDNs = dataBlocks + parityBlocks + 10;
> @Test(timeout = 24)
> public void testFileDecommission() throws Exception {
> LOG.info("Starting test testFileDecommission");
> final Path ecFile = new Path(ecDir, "testFileDecommission");
> int writeBytes = cellSize * dataBlocks;
> writeStripedFile(dfs, ecFile, writeBytes);
> Assert.assertEquals(0, bm.numOfUnderReplicatedBlocks());
> FileChecksum fileChecksum1 = dfs.getFileChecksum(ecFile, writeBytes);
> final INodeFile fileNode = cluster.getNamesystem().getFSDirectory()
> .getINode4Write(ecFile.toString()).asFile();
> LocatedBlocks locatedBlocks =
> StripedFileTestUtil.getLocatedBlocks(ecFile, dfs);
> LocatedBlock lb = dfs.getClient().getLocatedBlocks(ecFile.toString(), 0)
> .get(0);
> DatanodeInfo[] dnLocs = lb.getLocations();
> LocatedStripedBlock lastBlock =
> (LocatedStripedBlock)locatedBlocks.getLastLocatedBlock();
> DatanodeInfo[] storageInfos = lastBlock.getLocations();
> //
> DatanodeDescriptor datanodeDescriptor =
> cluster.getNameNode().getNamesystem()
>
> .getBlockManager().getDatanodeManager().getDatanode(storageInfos[6].getDatanodeUuid());
> BlockInfo firstBlock = fileNode.getBlocks()[0];
> DatanodeStorageInfo[] dStorageInfos = bm.getStorages(firstBlock);
> // the first heartbeat will consume 3 replica tasks
> for (int i = 0; i <= replicationStreamsHardLimit + 3; i++) {
> BlockManagerTestUtil.addBlockToBeReplicated(datanodeDescriptor, new
> Block(i),
> new DatanodeStorageInfo[]{dStorageInfos[0]});
> }
> assertEquals(dataBlocks + parityBlocks, dnLocs.length);
> int[] decommNodeIndex = {3, 4};
> final List decommisionNodes = new ArrayList();
> // add the node which will be decommissioning
> decommisionNodes.add(dnLocs[decommNodeIndex[0]]);
> decommisionNodes.add(dnLocs[decommNodeIndex[1]]);
> decommissionNode(0, decommisionNodes, AdminStates.DECOMMISSIONED);
> assertEquals(decommisionNodes.size(), fsn.getNumDecomLiveDataNodes());
> bm.getDatanodeManager().removeDatanode(datanodeDescriptor);
> //assertNull(checkFile(dfs, ecFile, 9, decommisionNodes, numDNs));
> // Ensure decommissioned datanode is not
[jira] [Commented] (HDFS-14845) Request is a replay (34) error in httpfs
[
https://issues.apache.org/jira/browse/HDFS-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931010#comment-16931010
]
Akira Ajisaka commented on HDFS-14845:
--
[~Prabhu Joseph] Thank you for finding the root cause and providing the
workaround!
[~eyang] We only want HttpFSAuthenticationFilter in HttpFS because HttpFS
doesn't have Web UI. We would like to use the same config as possible, so we
set JWTRedirectAuthenticationFilter for the other components that have Web UI
and that worked fine even in HttpFS before HADOOP-16314.
bq. If httpfs.authentication.* are not defined, then fall back to the default
behavior to be consistent with hadoop.http.authentication.type.
Now I'm +1 for Eric's suggestion. I'm thinking it's better that we can use the
common settings (hadoop.http.authentication.type) in HttpFS to avoid
component-specific settings.
> Request is a replay (34) error in httpfs
>
>
> Key: HDFS-14845
> URL: https://issues.apache.org/jira/browse/HDFS-14845
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: httpfs
>Affects Versions: 3.3.0
> Environment: Kerberos and ZKDelgationTokenSecretManager enabled in
> HttpFS
>Reporter: Akira Ajisaka
>Assignee: Prabhu Joseph
>Priority: Critical
> Attachments: HDFS-14845-001.patch
>
>
> We are facing "Request is a replay (34)" error when accessing to HDFS via
> httpfs on trunk.
> {noformat}
> % curl -i --negotiate -u : "https://:4443/webhdfs/v1/?op=liststatus"
> HTTP/1.1 401 Authentication required
> Date: Mon, 09 Sep 2019 06:00:04 GMT
> Date: Mon, 09 Sep 2019 06:00:04 GMT
> Pragma: no-cache
> X-Content-Type-Options: nosniff
> X-XSS-Protection: 1; mode=block
> WWW-Authenticate: Negotiate
> Set-Cookie: hadoop.auth=; Path=/; Secure; HttpOnly
> Cache-Control: must-revalidate,no-cache,no-store
> Content-Type: text/html;charset=iso-8859-1
> Content-Length: 271
> HTTP/1.1 403 GSSException: Failure unspecified at GSS-API level (Mechanism
> level: Request is a replay (34))
> Date: Mon, 09 Sep 2019 06:00:04 GMT
> Date: Mon, 09 Sep 2019 06:00:04 GMT
> Pragma: no-cache
> X-Content-Type-Options: nosniff
> X-XSS-Protection: 1; mode=block
> (snip)
> Set-Cookie: hadoop.auth=; Path=/; Secure; HttpOnly
> Cache-Control: must-revalidate,no-cache,no-store
> Content-Type: text/html;charset=iso-8859-1
> Content-Length: 413
>
>
>
> Error 403 GSSException: Failure unspecified at GSS-API level
> (Mechanism level: Request is a replay (34))
>
> HTTP ERROR 403
> Problem accessing /webhdfs/v1/. Reason:
> GSSException: Failure unspecified at GSS-API level (Mechanism level:
> Request is a replay (34))
>
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14768) In some cases, erasure blocks are corruption when they are reconstruct.
[
https://issues.apache.org/jira/browse/HDFS-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931008#comment-16931008
]
guojh commented on HDFS-14768:
--
[~surendrasingh] [~zhaoyim] Used the latest code in branch trunk still have the
problem. The UT I put in the Description can not passed.
> In some cases, erasure blocks are corruption when they are reconstruct.
>
>
> Key: HDFS-14768
> URL: https://issues.apache.org/jira/browse/HDFS-14768
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, erasure-coding, hdfs, namenode
>Affects Versions: 3.0.2
>Reporter: guojh
>Assignee: guojh
>Priority: Major
> Labels: patch
> Fix For: 3.3.0
>
> Attachments: 1568275810244.jpg, 1568276338275.jpg,
> HDFS-14768.000.patch, guojh_UT_after_deomission.txt,
> guojh_UT_before_deomission.txt, zhaoyiming_UT_after_deomission.txt,
> zhaoyiming_UT_beofre_deomission.txt
>
>
> Policy is RS-6-3-1024K, version is hadoop 3.0.2;
> We suppose a file's block Index is [0,1,2,3,4,5,6,7,8], And decommission
> index[3,4], increase the index 6 datanode's
> pendingReplicationWithoutTargets that make it large than
> replicationStreamsHardLimit(we set 14). Then, After the method
> chooseSourceDatanodes of BlockMananger, the liveBlockIndices is
> [0,1,2,3,4,5,7,8], Block Counter is, Live:7, Decommission:2.
> In method scheduleReconstruction of BlockManager, the additionalReplRequired
> is 9 - 7 = 2. After Namenode choose two target Datanode, will assign a
> erasureCode task to target datanode.
> When datanode get the task will build targetIndices from liveBlockIndices
> and target length. the code is blow.
> {code:java}
> // code placeholder
> targetIndices = new short[targets.length];
> private void initTargetIndices() {
> BitSet bitset = reconstructor.getLiveBitSet();
> int m = 0; hasValidTargets = false;
> for (int i = 0; i < dataBlkNum + parityBlkNum; i++) {
> if (!bitset.get) {
> if (reconstructor.getBlockLen > 0) {
> if (m < targets.length) {
> targetIndices[m++] = (short)i;
> hasValidTargets = true;
> }
> }
> }
> }
> {code}
> targetIndices[0]=6, and targetIndices[1] is aways 0 from initial value.
> The StripedReader is aways create reader from first 6 index block, and is
> [0,1,2,3,4,5]
> Use the index [0,1,2,3,4,5] to build target index[6,0] will trigger the isal
> bug. the block index6's data is corruption(all data is zero).
> I write a unit test can stabilize repreduce.
> {code:java}
> // code placeholder
> private int replicationStreamsHardLimit =
> DFSConfigKeys.DFS_NAMENODE_REPLICATION_STREAMS_HARD_LIMIT_DEFAULT;
> numDNs = dataBlocks + parityBlocks + 10;
> @Test(timeout = 24)
> public void testFileDecommission() throws Exception {
> LOG.info("Starting test testFileDecommission");
> final Path ecFile = new Path(ecDir, "testFileDecommission");
> int writeBytes = cellSize * dataBlocks;
> writeStripedFile(dfs, ecFile, writeBytes);
> Assert.assertEquals(0, bm.numOfUnderReplicatedBlocks());
> FileChecksum fileChecksum1 = dfs.getFileChecksum(ecFile, writeBytes);
> final INodeFile fileNode = cluster.getNamesystem().getFSDirectory()
> .getINode4Write(ecFile.toString()).asFile();
> LocatedBlocks locatedBlocks =
> StripedFileTestUtil.getLocatedBlocks(ecFile, dfs);
> LocatedBlock lb = dfs.getClient().getLocatedBlocks(ecFile.toString(), 0)
> .get(0);
> DatanodeInfo[] dnLocs = lb.getLocations();
> LocatedStripedBlock lastBlock =
> (LocatedStripedBlock)locatedBlocks.getLastLocatedBlock();
> DatanodeInfo[] storageInfos = lastBlock.getLocations();
> //
> DatanodeDescriptor datanodeDescriptor =
> cluster.getNameNode().getNamesystem()
>
> .getBlockManager().getDatanodeManager().getDatanode(storageInfos[6].getDatanodeUuid());
> BlockInfo firstBlock = fileNode.getBlocks()[0];
> DatanodeStorageInfo[] dStorageInfos = bm.getStorages(firstBlock);
> // the first heartbeat will consume 3 replica tasks
> for (int i = 0; i <= replicationStreamsHardLimit + 3; i++) {
> BlockManagerTestUtil.addBlockToBeReplicated(datanodeDescriptor, new
> Block(i),
> new DatanodeStorageInfo[]{dStorageInfos[0]});
> }
> assertEquals(dataBlocks + parityBlocks, dnLocs.length);
> int[] decommNodeIndex = {3, 4};
> final List decommisionNodes = new ArrayList();
> // add the node which will be decommissioning
> decommisionNodes.add(dnLocs[decommNodeIndex[0]]);
> decommisionNodes.add(dnLocs[decommNodeIndex[1]]);
> decommissionNode(0, decommisionNodes, AdminStates.DECOMMISSIONED);
> assertEquals(decommisionNodes.size(), fsn.getNumDecomLiveDataNodes());
> bm.getDatanodeManager().removeDatanode(datanodeDescriptor);
>
[jira] [Commented] (HDFS-14818) Check native pmdk lib by 'hadoop checknative' command
[ https://issues.apache.org/jira/browse/HDFS-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931007#comment-16931007 ] Feilong He commented on HDFS-14818: --- [^HDFS-14818.001.patch] has been uploaded with adding some comments for PMDK support states. > Check native pmdk lib by 'hadoop checknative' command > - > > Key: HDFS-14818 > URL: https://issues.apache.org/jira/browse/HDFS-14818 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: native >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Attachments: HDFS-14818.000.patch, HDFS-14818.001.patch, > check_native_after_building_with_PMDK.png, > check_native_after_building_with_PMDK_using_NAME_instead_of_REALPATH.png, > check_native_after_building_without_PMDK.png > > > Currently, 'hadoop checknative' command supports checking native libs, such > as zlib, snappy, openssl and ISA-L etc. It's necessary to include pmdk lib in > the checking. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2139) Update BeanUtils and Jackson Databind dependency versions
[ https://issues.apache.org/jira/browse/HDDS-2139?focusedWorklogId=313423=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313423 ] ASF GitHub Bot logged work on HDDS-2139: Author: ASF GitHub Bot Created on: 17/Sep/19 01:51 Start Date: 17/Sep/19 01:51 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1456: HDDS-2139. Update BeanUtils and Jackson Databind dependency versions. URL: https://github.com/apache/hadoop/pull/1456#issuecomment-532024260 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 41 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | ||| _ trunk Compile Tests _ | | +1 | shadedclient | 760 | branch has no errors when building and testing our client artifacts. | ||| _ Patch Compile Tests _ | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 750 | patch has no errors when building and testing our client artifacts. | ||| _ Other Tests _ | | +1 | asflicense | 35 | The patch does not generate ASF License warnings. | | | | 1696 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1456/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1456 | | Optional Tests | dupname asflicense xml | | uname | Linux 72c3ca204568 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 2358e53 | | Max. process+thread count | 411 (vs. ulimit of 5500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1456/1/console | | versions | git=2.7.4 maven=3.3.9 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313423) Time Spent: 20m (was: 10m) > Update BeanUtils and Jackson Databind dependency versions > - > > Key: HDDS-2139 > URL: https://issues.apache.org/jira/browse/HDDS-2139 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The following Ozone dependencies have known security vulnerabilities. We > should update them to newer/ latest versions. > * Apache Common BeanUtils version 1.9.3 > * Fasterxml Jackson version 2.9.5 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14818) Check native pmdk lib by 'hadoop checknative' command
[
https://issues.apache.org/jira/browse/HDFS-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930216#comment-16930216
]
Feilong He edited comment on HDFS-14818 at 9/17/19 1:50 AM:
Thanks [~rakeshr] for your comments.
To make the code change effect clear to reviewers, I posted some screenshots.
* The below picture shows the result of 'hadoop checknative' afer building
WITH PMDK. The build cmd is 'mvn clean package -Pdist,native -DskipTests -Dtar
-Drequire.pmdk'.
!check_native_after_building_with_PMDK.png!
* The below picture shows the result of 'hadoop checknative' afer building
WITHOUT PMDK. The build cmd is 'mvn clean package -Pdist,native -DskipTests
-Dtar'.
!check_native_after_building_without_PMDK.png!
* The below picture shows the result of 'hadoop checknative' afer building
WITH PMDK, but shading the modification brought by this patch for
CMakeLists.txt, i.e., still use 'NAME' instead of 'REALPATH'. The build cmd is
'mvn clean package -Pdist,native -DskipTests -Dtar -Drequire.pmdk'.
!check_native_after_building_with_PMDK_using_NAME_instead_of_REALPATH.png!
{quote}{{SupportState.PMDK_LIB_NOT_FOUND}} - its unused now, can you remove it.
{quote}
In some env, if the PMDK native lib is not found, this state and its message
will help user to identify the fact. So I am leaning to keep this state.
{quote}Any reason to change 'NAME' to 'REALPATH'.
{quote}
As the above 3rd picture shows, if 'NAME' is used instead of ‘REALPATH', only
the lib name can be obtained and then printed by 'hadoop checknative'. In this
patch, by using 'REALPATH', the real path of the target lib will be kept, which
is more useful to user, I think.
Please refer to
[https://cmake.org/cmake/help/v3.15/command/get_filename_component.html].
was (Author: philohe):
Thanks [~rakeshr] for your comments.
To make the code change effect clear to reviewers, I posted some screenshots.
* The below picture shows the result of 'hadoop checknative' afer building
WITH PMDK. The build cmd is 'mvn clean package -Pdist,native -DskipTests -Dtar
-Drequire.pmdk'.
!check_native_after_building_with_PMDK.png!
* The below picture shows the result of 'hadoop checknative' afer building
WITHOUT PMDK. The build cmd is 'mvn clean package -Pdist,native -DskipTests
-Dtar'.
!check_native_after_building_without_PMDK.png!
* The below picture shows the result of 'hadoop checknative' afer building
WITH PMDK, but shading the modification brought by this patch for
CMakeLists.txt, i.e., still use 'NAME' instead of 'REALPATH'. The build cmd is
'mvn clean package -Pdist,native -DskipTests -Dtar -Drequire.pmdk'.
!check_native_after_building_with_PMDK_using_NAME_instead_of_REALPATH.png!
{quote}{{SupportState.PMDK_LIB_NOT_FOUND}} - its unused now, can you remove it.
{quote}
In some env, if the PMDK native lib is nout found, this state and its message
will help user to identify the fact. So I am leaning to keep this state.
{quote}Any reason to change 'NAME' to 'REALPATH'.
{quote}
As the above 3rd picture shows, if 'NAME' is used instead of ‘REALPATH', only
the lib name can be obtained and then printed by 'hadoop checknative'. In this
patch, by using 'REALPATH', the real path of the target lib will be kept, which
is more useful to user, I think.
Please refer to
[https://cmake.org/cmake/help/v3.15/command/get_filename_component.html].
> Check native pmdk lib by 'hadoop checknative' command
> -
>
> Key: HDFS-14818
> URL: https://issues.apache.org/jira/browse/HDFS-14818
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: native
>Reporter: Feilong He
>Assignee: Feilong He
>Priority: Major
> Attachments: HDFS-14818.000.patch, HDFS-14818.001.patch,
> check_native_after_building_with_PMDK.png,
> check_native_after_building_with_PMDK_using_NAME_instead_of_REALPATH.png,
> check_native_after_building_without_PMDK.png
>
>
> Currently, 'hadoop checknative' command supports checking native libs, such
> as zlib, snappy, openssl and ISA-L etc. It's necessary to include pmdk lib in
> the checking.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14818) Check native pmdk lib by 'hadoop checknative' command
[ https://issues.apache.org/jira/browse/HDFS-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feilong He updated HDFS-14818: -- Attachment: HDFS-14818.001.patch > Check native pmdk lib by 'hadoop checknative' command > - > > Key: HDFS-14818 > URL: https://issues.apache.org/jira/browse/HDFS-14818 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: native >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Attachments: HDFS-14818.000.patch, HDFS-14818.001.patch, > check_native_after_building_with_PMDK.png, > check_native_after_building_with_PMDK_using_NAME_instead_of_REALPATH.png, > check_native_after_building_without_PMDK.png > > > Currently, 'hadoop checknative' command supports checking native libs, such > as zlib, snappy, openssl and ISA-L etc. It's necessary to include pmdk lib in > the checking. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6524) Choosing datanode retries times considering with block replica number
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930997#comment-16930997 ] Lisheng Sun commented on HDFS-6524: --- hi [~elgoiri] , i agree your option. There are same problems in TestRollingUpgrade and TestEncryptedTransfer. we can fix them by putting up new jira later. > Choosing datanode retries times considering with block replica number > -- > > Key: HDFS-6524 > URL: https://issues.apache.org/jira/browse/HDFS-6524 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0-alpha1 >Reporter: Liang Xie >Assignee: Lisheng Sun >Priority: Minor > Labels: BB2015-05-TBR > Attachments: HDFS-6524.001.patch, HDFS-6524.002.patch, > HDFS-6524.003.patch, HDFS-6524.004.patch, HDFS-6524.005.patch, HDFS-6524.txt > > > Currently the chooseDataNode() does retry with the setting: > dfsClientConf.maxBlockAcquireFailures, which by default is 3 > (DFS_CLIENT_MAX_BLOCK_ACQUIRE_FAILURES_DEFAULT = 3), it would be better > having another option, block replication factor. One cluster with only two > block replica setting, or using Reed-solomon encoding solution with one > replica factor. It helps to reduce the long tail latency. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6524) Choosing datanode retries times considering with block replica number
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-6524: -- Attachment: HDFS-6524.005.patch > Choosing datanode retries times considering with block replica number > -- > > Key: HDFS-6524 > URL: https://issues.apache.org/jira/browse/HDFS-6524 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0-alpha1 >Reporter: Liang Xie >Assignee: Lisheng Sun >Priority: Minor > Labels: BB2015-05-TBR > Attachments: HDFS-6524.001.patch, HDFS-6524.002.patch, > HDFS-6524.003.patch, HDFS-6524.004.patch, HDFS-6524.005.patch, HDFS-6524.txt > > > Currently the chooseDataNode() does retry with the setting: > dfsClientConf.maxBlockAcquireFailures, which by default is 3 > (DFS_CLIENT_MAX_BLOCK_ACQUIRE_FAILURES_DEFAULT = 3), it would be better > having another option, block replication factor. One cluster with only two > block replica setting, or using Reed-solomon encoding solution with one > replica factor. It helps to reduce the long tail latency. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2139) Update BeanUtils and Jackson Databind dependency versions
[ https://issues.apache.org/jira/browse/HDDS-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2139: - Labels: pull-request-available (was: ) > Update BeanUtils and Jackson Databind dependency versions > - > > Key: HDDS-2139 > URL: https://issues.apache.org/jira/browse/HDDS-2139 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > > The following Ozone dependencies have known security vulnerabilities. We > should update them to newer/ latest versions. > * Apache Common BeanUtils version 1.9.3 > * Fasterxml Jackson version 2.9.5 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2139) Update BeanUtils and Jackson Databind dependency versions
[ https://issues.apache.org/jira/browse/HDDS-2139?focusedWorklogId=313401=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313401 ] ASF GitHub Bot logged work on HDDS-2139: Author: ASF GitHub Bot Created on: 17/Sep/19 01:22 Start Date: 17/Sep/19 01:22 Worklog Time Spent: 10m Work Description: hanishakoneru commented on pull request #1456: HDDS-2139. Update BeanUtils and Jackson Databind dependency versions. URL: https://github.com/apache/hadoop/pull/1456 The following Ozone dependencies have known security vulnerabilities. We should update them to newer/ latest versions. - Apache Common BeanUtils version 1.9.3 - Fasterxml Jackson version 2.9.5 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313401) Remaining Estimate: 0h Time Spent: 10m > Update BeanUtils and Jackson Databind dependency versions > - > > Key: HDDS-2139 > URL: https://issues.apache.org/jira/browse/HDDS-2139 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The following Ozone dependencies have known security vulnerabilities. We > should update them to newer/ latest versions. > * Apache Common BeanUtils version 1.9.3 > * Fasterxml Jackson version 2.9.5 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2139) Update BeanUtils and Jackson Databind dependency versions
[ https://issues.apache.org/jira/browse/HDDS-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-2139: - Description: The following Ozone dependencies have known security vulnerabilities. We should update them to newer/ latest versions. * Apache Common BeanUtils version 1.9.3 * Fasterxml Jackson version 2.9.5 was: The following Ozone dependencies have known security vulnerabilities. We should update them to newer/ latest versions. * Apache Common BeanUtils version 1.9.3 * Fasterxml Jackson version 2.9.5 > Update BeanUtils and Jackson Databind dependency versions > - > > Key: HDDS-2139 > URL: https://issues.apache.org/jira/browse/HDDS-2139 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > > The following Ozone dependencies have known security vulnerabilities. We > should update them to newer/ latest versions. > * Apache Common BeanUtils version 1.9.3 > * Fasterxml Jackson version 2.9.5 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14851) WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks
[ https://issues.apache.org/jira/browse/HDFS-14851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930994#comment-16930994 ] Íñigo Goiri commented on HDFS-14851: [~crh], you guys use WebHDFS right? Have you found anything like this? > WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks > - > > Key: HDFS-14851 > URL: https://issues.apache.org/jira/browse/HDFS-14851 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Danny Becker >Assignee: Danny Becker >Priority: Minor > Attachments: HDFS-14851.001.patch > > > WebHdfs returns 200 status code for Open operations on files with missing or > corrupt blocks. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2139) Update BeanUtils and Jackson Databind dependency versions
Hanisha Koneru created HDDS-2139: Summary: Update BeanUtils and Jackson Databind dependency versions Key: HDDS-2139 URL: https://issues.apache.org/jira/browse/HDDS-2139 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Hanisha Koneru Assignee: Hanisha Koneru The following Ozone dependencies have known security vulnerabilities. We should update them to newer/ latest versions. * Apache Common BeanUtils version 1.9.3 * Fasterxml Jackson version 2.9.5 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14174) Enhance Audit for chown ( internally setOwner)
[ https://issues.apache.org/jira/browse/HDFS-14174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930967#comment-16930967 ] Wei-Chiu Chuang commented on HDFS-14174: I just wanted to emphasize there are tools that parse audit log files. If you can make it backward compatible (say, adding the existing owner information to the end of audit log), I am all for it. > Enhance Audit for chown ( internally setOwner) > - > > Key: HDFS-14174 > URL: https://issues.apache.org/jira/browse/HDFS-14174 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.6.0 >Reporter: Sailesh Patel >Assignee: hemanthboyina >Priority: Minor > > When a hdfs dfs -chown command is executed, the audit log does not capture > the existing owner and the new owner. > Need to capture the old and new owner to allow auditing to be effective > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14771) Backport HDFS-14617 to branch-2 (Improve fsimage load time by writing sub-sections to the fsimage index)
[ https://issues.apache.org/jira/browse/HDFS-14771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930951#comment-16930951 ] Wei-Chiu Chuang commented on HDFS-14771: +1 Once this features goes into branch-2 (which will become 2.10 release), and that the feature is enabled, I assume the only upgrade path is 2.10 --> 3.3, correct? > Backport HDFS-14617 to branch-2 (Improve fsimage load time by writing > sub-sections to the fsimage index) > > > Key: HDFS-14771 > URL: https://issues.apache.org/jira/browse/HDFS-14771 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 2.10.0 >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Labels: release-blocker > Attachments: HDFS-14771.branch-2.001.patch, > HDFS-14771.branch-2.002.patch, HDFS-14771.branch-2.003.patch > > > This JIRA aims to backport HDFS-14617 to branch-2: fsimage load time by > writing sub-sections to the fsimage index. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14851) WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks
[ https://issues.apache.org/jira/browse/HDFS-14851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Becker reassigned HDFS-14851: --- Assignee: Danny Becker > WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks > - > > Key: HDFS-14851 > URL: https://issues.apache.org/jira/browse/HDFS-14851 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Danny Becker >Assignee: Danny Becker >Priority: Minor > > WebHdfs returns 200 status code for Open operations on files with missing or > corrupt blocks. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14851) WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks
[ https://issues.apache.org/jira/browse/HDFS-14851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Becker updated HDFS-14851: Attachment: HDFS-14851.001.patch > WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks > - > > Key: HDFS-14851 > URL: https://issues.apache.org/jira/browse/HDFS-14851 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Danny Becker >Assignee: Danny Becker >Priority: Minor > Attachments: HDFS-14851.001.patch > > > WebHdfs returns 200 status code for Open operations on files with missing or > corrupt blocks. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14851) WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks
Danny Becker created HDFS-14851: --- Summary: WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks Key: HDFS-14851 URL: https://issues.apache.org/jira/browse/HDFS-14851 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Reporter: Danny Becker WebHdfs returns 200 status code for Open operations on files with missing or corrupt blocks. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2111) XSS fragments can be injected to the S3g landing page
[
https://issues.apache.org/jira/browse/HDDS-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDDS-2111:
---
Fix Version/s: 0.5.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> XSS fragments can be injected to the S3g landing page
> ---
>
> Key: HDDS-2111
> URL: https://issues.apache.org/jira/browse/HDDS-2111
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Aayush
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> VULNERABILITY DETAILS
> There is a way to bypass anti-XSS filter for DOM XSS exploiting a
> "window.location.href".
> Considering a typical URL:
> scheme://domain:port/path?query_string#fragment_id
> Browsers encode correctly both "path" and "query_string", but not the
> "fragment_id".
> So if used "fragment_id" the vector is also not logged on Web Server.
> VERSION
> Chrome Version: 10.0.648.134 (Official Build 77917) beta
> REPRODUCTION CASE
> This is an index.html page:
> {code:java}
> aws s3api --endpoint
> document.write(window.location.href.replace("static/", ""))
> create-bucket --bucket=wordcount
> {code}
> The attack vector is:
> index.html?#alert('XSS');
> * PoC:
> For your convenience, a minimalist PoC is located on:
> http://security.onofri.org/xss_location.html?#alert('XSS');
> * References
> - DOM Based Cross-Site Scripting or XSS of the Third Kind -
> http://www.webappsec.org/projects/articles/071105.shtml
> reference:-
> https://bugs.chromium.org/p/chromium/issues/detail?id=76796
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2111) XSS fragments can be injected to the S3g landing page
[
https://issues.apache.org/jira/browse/HDDS-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930939#comment-16930939
]
Anu Engineer commented on HDDS-2111:
Thank you [~adeo] for reporting this issue. [~elek] Thank you for fixing this
issue. I have committed this patch to the trunk branch.
FYI: [~nandakumar131] You might want to cherry-pick this to 0.4.1
> XSS fragments can be injected to the S3g landing page
> ---
>
> Key: HDDS-2111
> URL: https://issues.apache.org/jira/browse/HDDS-2111
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Aayush
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> VULNERABILITY DETAILS
> There is a way to bypass anti-XSS filter for DOM XSS exploiting a
> "window.location.href".
> Considering a typical URL:
> scheme://domain:port/path?query_string#fragment_id
> Browsers encode correctly both "path" and "query_string", but not the
> "fragment_id".
> So if used "fragment_id" the vector is also not logged on Web Server.
> VERSION
> Chrome Version: 10.0.648.134 (Official Build 77917) beta
> REPRODUCTION CASE
> This is an index.html page:
> {code:java}
> aws s3api --endpoint
> document.write(window.location.href.replace("static/", ""))
> create-bucket --bucket=wordcount
> {code}
> The attack vector is:
> index.html?#alert('XSS');
> * PoC:
> For your convenience, a minimalist PoC is located on:
> http://security.onofri.org/xss_location.html?#alert('XSS');
> * References
> - DOM Based Cross-Site Scripting or XSS of the Third Kind -
> http://www.webappsec.org/projects/articles/071105.shtml
> reference:-
> https://bugs.chromium.org/p/chromium/issues/detail?id=76796
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2111) XSS fragments can be injected to the S3g landing page
[
https://issues.apache.org/jira/browse/HDDS-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930938#comment-16930938
]
Hudson commented on HDDS-2111:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17310 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17310/])
HDDS-2111. XSS fragments can be injected to the S3g landing page (aengineer:
rev 2358e53e9c9f8489b24648b1017eb856d4bd42b0)
* (edit) hadoop-ozone/s3gateway/src/main/resources/webapps/static/index.html
* (add) hadoop-ozone/s3gateway/src/main/resources/webapps/static/s3g.js
> XSS fragments can be injected to the S3g landing page
> ---
>
> Key: HDDS-2111
> URL: https://issues.apache.org/jira/browse/HDDS-2111
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Aayush
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> VULNERABILITY DETAILS
> There is a way to bypass anti-XSS filter for DOM XSS exploiting a
> "window.location.href".
> Considering a typical URL:
> scheme://domain:port/path?query_string#fragment_id
> Browsers encode correctly both "path" and "query_string", but not the
> "fragment_id".
> So if used "fragment_id" the vector is also not logged on Web Server.
> VERSION
> Chrome Version: 10.0.648.134 (Official Build 77917) beta
> REPRODUCTION CASE
> This is an index.html page:
> {code:java}
> aws s3api --endpoint
> document.write(window.location.href.replace("static/", ""))
> create-bucket --bucket=wordcount
> {code}
> The attack vector is:
> index.html?#alert('XSS');
> * PoC:
> For your convenience, a minimalist PoC is located on:
> http://security.onofri.org/xss_location.html?#alert('XSS');
> * References
> - DOM Based Cross-Site Scripting or XSS of the Third Kind -
> http://www.webappsec.org/projects/articles/071105.shtml
> reference:-
> https://bugs.chromium.org/p/chromium/issues/detail?id=76796
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2111) XSS fragments can be injected to the S3g landing page
[
https://issues.apache.org/jira/browse/HDDS-2111?focusedWorklogId=313341=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313341
]
ASF GitHub Bot logged work on HDDS-2111:
Author: ASF GitHub Bot
Created on: 16/Sep/19 22:49
Start Date: 16/Sep/19 22:49
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #1447: HDDS-2111.
XSS fragments can be injected to the S3g landing page
URL: https://github.com/apache/hadoop/pull/1447
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313341)
Time Spent: 50m (was: 40m)
> XSS fragments can be injected to the S3g landing page
> ---
>
> Key: HDDS-2111
> URL: https://issues.apache.org/jira/browse/HDDS-2111
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Aayush
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> VULNERABILITY DETAILS
> There is a way to bypass anti-XSS filter for DOM XSS exploiting a
> "window.location.href".
> Considering a typical URL:
> scheme://domain:port/path?query_string#fragment_id
> Browsers encode correctly both "path" and "query_string", but not the
> "fragment_id".
> So if used "fragment_id" the vector is also not logged on Web Server.
> VERSION
> Chrome Version: 10.0.648.134 (Official Build 77917) beta
> REPRODUCTION CASE
> This is an index.html page:
> {code:java}
> aws s3api --endpoint
> document.write(window.location.href.replace("static/", ""))
> create-bucket --bucket=wordcount
> {code}
> The attack vector is:
> index.html?#alert('XSS');
> * PoC:
> For your convenience, a minimalist PoC is located on:
> http://security.onofri.org/xss_location.html?#alert('XSS');
> * References
> - DOM Based Cross-Site Scripting or XSS of the Third Kind -
> http://www.webappsec.org/projects/articles/071105.shtml
> reference:-
> https://bugs.chromium.org/p/chromium/issues/detail?id=76796
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[
https://issues.apache.org/jira/browse/HDDS-2114?focusedWorklogId=313338=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313338
]
ASF GitHub Bot logged work on HDDS-2114:
Author: ASF GitHub Bot
Created on: 16/Sep/19 22:37
Start Date: 16/Sep/19 22:37
Worklog Time Spent: 10m
Work Description: anuengineer commented on issue #1440: HDDS-2114: Rename
does not preserve non-explicitly created interim directories
URL: https://github.com/apache/hadoop/pull/1440#issuecomment-531984790
I am going to +1 this. Since we want to make sure Hive works.
I just want to understand this more clearly. The issue is really that if we
were a real file system, then there is nothing called an implicit path. Since
we are an object store, there is a notion of a implicitly created file path (in
this case the intermediary directories). I am guessing that S3AFS has the same
problem, and either Hive has a workaround for this, or S3A is doing something
clever. Do we know how Hive works on S3?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313338)
Time Spent: 1h 10m (was: 1h)
> Rename does not preserve non-explicitly created interim directories
> ---
>
> Key: HDDS-2114
> URL: https://issues.apache.org/jira/browse/HDDS-2114
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Lokesh Jain
>Priority: Critical
> Labels: pull-request-available
> Attachments: demonstrative_test.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> I am attaching a patch that adds a test that demonstrates the problem.
> The scenario is coming from the way how Hive implements acid transactions
> with the ORC table format, but the test is redacted to the simplest possible
> code that reproduces the issue.
> The scenario:
> * Given a 3 level directory structure, where the top level directory was
> explicitly created, and the interim directory is implicitly created (for
> example either by creating a file with create("/top/interim/file") or by
> creating a directory with mkdirs("top/interim/dir"))
> * When the leaf is moved out from the implicitly created directory making
> this directory an empty directory
> * Then a FileNotFoundException is thrown when getFileStatus or listStatus is
> called on the interim directory.
> The expected behaviour:
> after the directory is becoming empty, the directory should still be part of
> the file system, moreover an empty FileStatus array should be returned when
> listStatus is called on it, and also a valid FileStatus object should be
> returned when getFileStatus is called on it.
>
>
> As this issue is present with Hive, and as this is how a FileSystem is
> expected to work this seems to be an at least critical issue as I see, please
> feel free to change the priority if needed.
> Also please note that, if the interim directory is explicitly created with
> mkdirs("top/interim") before creating the leaf, then the issue does not
> appear.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2114) Rename does not preserve non-explicitly created interim directories
[
https://issues.apache.org/jira/browse/HDDS-2114?focusedWorklogId=313337=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313337
]
ASF GitHub Bot logged work on HDDS-2114:
Author: ASF GitHub Bot
Created on: 16/Sep/19 22:36
Start Date: 16/Sep/19 22:36
Worklog Time Spent: 10m
Work Description: anuengineer commented on issue #1440: HDDS-2114: Rename
does not preserve non-explicitly created interim directories
URL: https://github.com/apache/hadoop/pull/1440#issuecomment-531984790
I am going to +1 this. Since we want to make sure Hive works.
I just want to understand this more clearly. The issue is really that if we
were a real file system, then there is nothing called an implicit path. Since
we are an object store, there is a notion of a implicitly created file system.
I am guess that S3AFS has the same problem, and either Hive has a workaround
for this, or S3A is doing something really clever. Do we know how Hive works on
S3?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313337)
Time Spent: 1h (was: 50m)
> Rename does not preserve non-explicitly created interim directories
> ---
>
> Key: HDDS-2114
> URL: https://issues.apache.org/jira/browse/HDDS-2114
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Istvan Fajth
>Assignee: Lokesh Jain
>Priority: Critical
> Labels: pull-request-available
> Attachments: demonstrative_test.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> I am attaching a patch that adds a test that demonstrates the problem.
> The scenario is coming from the way how Hive implements acid transactions
> with the ORC table format, but the test is redacted to the simplest possible
> code that reproduces the issue.
> The scenario:
> * Given a 3 level directory structure, where the top level directory was
> explicitly created, and the interim directory is implicitly created (for
> example either by creating a file with create("/top/interim/file") or by
> creating a directory with mkdirs("top/interim/dir"))
> * When the leaf is moved out from the implicitly created directory making
> this directory an empty directory
> * Then a FileNotFoundException is thrown when getFileStatus or listStatus is
> called on the interim directory.
> The expected behaviour:
> after the directory is becoming empty, the directory should still be part of
> the file system, moreover an empty FileStatus array should be returned when
> listStatus is called on it, and also a valid FileStatus object should be
> returned when getFileStatus is called on it.
>
>
> As this issue is present with Hive, and as this is how a FileSystem is
> expected to work this seems to be an at least critical issue as I see, please
> feel free to change the priority if needed.
> Also please note that, if the interim directory is explicitly created with
> mkdirs("top/interim") before creating the leaf, then the issue does not
> appear.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2110) Arbitrary file can be downloaded with the help of ProfilerServlet
[
https://issues.apache.org/jira/browse/HDDS-2110?focusedWorklogId=313327=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313327
]
ASF GitHub Bot logged work on HDDS-2110:
Author: ASF GitHub Bot
Created on: 16/Sep/19 22:19
Start Date: 16/Sep/19 22:19
Worklog Time Spent: 10m
Work Description: anuengineer commented on issue #1448: HDDS-2110.
Arbitrary file can be downloaded with the help of ProfilerServlet
URL: https://github.com/apache/hadoop/pull/1448#issuecomment-531980439
Do you want to write a FindBugs Suppression rule with a pointer to
HDDS-2110, So that people know why we are suppressing the Findbugs warning?,
and also suppress this findbugs?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313327)
Time Spent: 50m (was: 40m)
> Arbitrary file can be downloaded with the help of ProfilerServlet
> -
>
> Key: HDDS-2110
> URL: https://issues.apache.org/jira/browse/HDDS-2110
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Native
>Reporter: Aayush
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The LOC 324 in the file
> [ProfileServlet.java|https://github.com/apache/hadoop/blob/217bdbd940a96986df3b96899b43caae2b5a9ed2/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/ProfileServlet.java]
> is prone to an arbitrary file download:-
> {code:java}
> protected void doGetDownload(String fileName, final HttpServletRequest req,
>final HttpServletResponse resp) throws IOException {
> File requestedFile =
> ProfileServlet.OUTPUT_DIR.resolve(fileName).toAbsolutePath().toFile();{code}
> As the String fileName is directly considered as the requested file.
>
> Which is called at LOC 180 with HTTP request directly passed:-
> {code:java}
> if (req.getParameter("file") != null) {
> doGetDownload(req.getParameter("file"), req, resp);
> return;
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2121) Create a shaded ozone filesystem (client) jar
[ https://issues.apache.org/jira/browse/HDDS-2121?focusedWorklogId=313321=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313321 ] ASF GitHub Bot logged work on HDDS-2121: Author: ASF GitHub Bot Created on: 16/Sep/19 22:02 Start Date: 16/Sep/19 22:02 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #1452: HDDS-2121. Create a shaded ozone filesystem (client) jar URL: https://github.com/apache/hadoop/pull/1452#discussion_r324905642 ## File path: hadoop-ozone/ozonefs-lib-current/pom.xml ## @@ -83,6 +63,78 @@ true + +org.apache.maven.plugins +maven-shade-plugin + + +package + + shade + + + + + classworlds:classworlds Review comment: What is this classworlds:classworlds, is this mistakenly added from maven example? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313321) Time Spent: 50m (was: 40m) > Create a shaded ozone filesystem (client) jar > - > > Key: HDDS-2121 > URL: https://issues.apache.org/jira/browse/HDDS-2121 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Arpit Agarwal >Assignee: Elek, Marton >Priority: Blocker > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > We need a shaded Ozonefs jar that does not include Hadoop ecosystem > components (Hadoop, HDFS, Ratis, Zookeeper). > A common expected use case for Ozone is Hadoop clients (3.2.0 and later) > wanting to access Ozone via the Ozone Filesystem interface. For these > clients, we want to add Ozone file system jar to the classpath, however we > want to use Hadoop ecosystem dependencies that are `provided` and already > expected to be in the client classpath. > Note that this is different from the legacy jar which bundles a shaded Hadoop > 3.2.0. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14836) FileIoProvider should not increase FileIoErrors metric in datanode volume metric
[
https://issues.apache.org/jira/browse/HDFS-14836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930860#comment-16930860
]
Hadoop QA commented on HDFS-14836:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 45s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:39e82acc485 |
| JIRA Issue | HDFS-14836 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12980337/HDFS-14836-trunk-001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e9144a219d91 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 56f042c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/27886/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/27886/testReport/ |
| Max. process+thread count | 3437 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Resolved] (HDDS-2098) Ozone shell command prints out ERROR when the log4j file is not present.
[
https://issues.apache.org/jira/browse/HDDS-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan resolved HDDS-2098.
-
Resolution: Won't Fix
This is no longer needed since we can handle it in the cluster management tool
(CM, Ambari etc) level.
> Ozone shell command prints out ERROR when the log4j file is not present.
>
>
> Key: HDDS-2098
> URL: https://issues.apache.org/jira/browse/HDDS-2098
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> *Exception Trace*
> {code}
> log4j:ERROR Could not read configuration file from URL
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> java.io.FileNotFoundException: /etc/ozone/conf/ozone-shell-log4j.properties
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.(FileInputStream.java:138)
> at java.io.FileInputStream.(FileInputStream.java:93)
> at
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66)
> at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
> at
> org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
> at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
> at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
> at org.apache.hadoop.ozone.web.ozShell.Shell.(Shell.java:35)
> log4j:ERROR Ignoring configuration file
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> log4j:WARN No appenders could be found for logger
> (io.jaegertracing.thrift.internal.senders.ThriftSenderFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> {
> "metadata" : { },
> "name" : "vol-test-putfile-1567740142",
> "admin" : "root",
> "owner" : "root",
> "creationTime" : 1567740146501,
> "acls" : [ {
> "type" : "USER",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ],
> "quota" : 1152921504606846976
> }
> {code}
> *Fix*
> When a log4j file is not present, the default should be console.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2135) OM Metric mismatch (MultipartUpload failures)
[
https://issues.apache.org/jira/browse/HDDS-2135?focusedWorklogId=313276=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313276
]
ASF GitHub Bot logged work on HDDS-2135:
Author: ASF GitHub Bot
Created on: 16/Sep/19 20:14
Start Date: 16/Sep/19 20:14
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on issue #1453: HDDS-2135. OM
Metric mismatch (MultipartUpload failures)
URL: https://github.com/apache/hadoop/pull/1453#issuecomment-531939480
/retest
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313276)
Time Spent: 1h (was: 50m)
> OM Metric mismatch (MultipartUpload failures)
> -
>
> Key: HDDS-2135
> URL: https://issues.apache.org/jira/browse/HDDS-2135
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Affects Versions: 0.4.1
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {{incNumCommitMultipartUploadPartFails()}} increments
> {{numInitiateMultipartUploadFails}} instead of the counter for commit
> failures.
> https://github.com/apache/hadoop/blob/85b1c728e4ed22f03db255f5ef34a2a79eb20d52/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMMetrics.java#L310-L312
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[
https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930833#comment-16930833
]
Hadoop QA commented on HDFS-14431:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red} HDFS-14431 does not apply to HDFS-13891. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14431 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12969060/HDFS-14431-HDFS-13891.007.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/27887/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Rename with multiple subclusters should fail if no eligible locations
> --
>
> Key: HDFS-14431
> URL: https://issues.apache.org/jira/browse/HDFS-14431
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-14431-HDFS-13891.001.patch,
> HDFS-14431-HDFS-13891.002.patch, HDFS-14431-HDFS-13891.003.patch,
> HDFS-14431-HDFS-13891.004.patch, HDFS-14431-HDFS-13891.005.patch,
> HDFS-14431-HDFS-13891.006.patch, HDFS-14431-HDFS-13891.007.patch
>
>
> Currently, the rename will fail with FileNotFoundException which is not clear
> to the user.
> The operation should fail stating the reason is that there are no eligible
> destinations.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14431) RBF: Rename with multiple subclusters should fail if no eligible locations
[ https://issues.apache.org/jira/browse/HDFS-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930830#comment-16930830 ] CR Hota commented on HDFS-14431: [~elgoiri] Many thanks for all the work done so far. Took a look at the patch and approach seems error prone as the operations in totality are NOT atomic. Filesystems are not transactional in nature. Since rename is very hard to get right, may I suggest we approach it as we did with some other features. Let's come up with a design doc and write down the issues, possible approaches and what all use cases we can solve and can't. We can all collaborate. Please count me in. For someone new, its very hard to get the context of what is being solved and what use cases are not. On a side note, with the lack of atomic renames here is how we are approaching renames in the short term. Most query engines (ex Hive) are equipped to handle rename failure by initiating a copy. In the scenario where rename is across clusters, hive is instructed to invoke a copy operation. FYI [~ayushtkn] [~xuzq_zander] > RBF: Rename with multiple subclusters should fail if no eligible locations > -- > > Key: HDFS-14431 > URL: https://issues.apache.org/jira/browse/HDFS-14431 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14431-HDFS-13891.001.patch, > HDFS-14431-HDFS-13891.002.patch, HDFS-14431-HDFS-13891.003.patch, > HDFS-14431-HDFS-13891.004.patch, HDFS-14431-HDFS-13891.005.patch, > HDFS-14431-HDFS-13891.006.patch, HDFS-14431-HDFS-13891.007.patch > > > Currently, the rename will fail with FileNotFoundException which is not clear > to the user. > The operation should fail stating the reason is that there are no eligible > destinations. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2098) Ozone shell command prints out ERROR when the log4j file is not present.
[
https://issues.apache.org/jira/browse/HDDS-2098?focusedWorklogId=313273=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313273
]
ASF GitHub Bot logged work on HDDS-2098:
Author: ASF GitHub Bot
Created on: 16/Sep/19 20:09
Start Date: 16/Sep/19 20:09
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on issue #1411: HDDS-2098 : Ozone
shell command prints out ERROR when the log4j file …
URL: https://github.com/apache/hadoop/pull/1411#issuecomment-531937463
Closing this out since we can handle it from the cluster management tool.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313273)
Time Spent: 1.5h (was: 1h 20m)
> Ozone shell command prints out ERROR when the log4j file is not present.
>
>
> Key: HDDS-2098
> URL: https://issues.apache.org/jira/browse/HDDS-2098
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> *Exception Trace*
> {code}
> log4j:ERROR Could not read configuration file from URL
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> java.io.FileNotFoundException: /etc/ozone/conf/ozone-shell-log4j.properties
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.(FileInputStream.java:138)
> at java.io.FileInputStream.(FileInputStream.java:93)
> at
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66)
> at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
> at
> org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
> at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
> at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
> at org.apache.hadoop.ozone.web.ozShell.Shell.(Shell.java:35)
> log4j:ERROR Ignoring configuration file
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> log4j:WARN No appenders could be found for logger
> (io.jaegertracing.thrift.internal.senders.ThriftSenderFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> {
> "metadata" : { },
> "name" : "vol-test-putfile-1567740142",
> "admin" : "root",
> "owner" : "root",
> "creationTime" : 1567740146501,
> "acls" : [ {
> "type" : "USER",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ],
> "quota" : 1152921504606846976
> }
> {code}
> *Fix*
> When a log4j file is not present, the default should be console.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2098) Ozone shell command prints out ERROR when the log4j file is not present.
[
https://issues.apache.org/jira/browse/HDDS-2098?focusedWorklogId=313274=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313274
]
ASF GitHub Bot logged work on HDDS-2098:
Author: ASF GitHub Bot
Created on: 16/Sep/19 20:09
Start Date: 16/Sep/19 20:09
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on pull request #1411: HDDS-2098
: Ozone shell command prints out ERROR when the log4j file …
URL: https://github.com/apache/hadoop/pull/1411
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313274)
Time Spent: 1h 40m (was: 1.5h)
> Ozone shell command prints out ERROR when the log4j file is not present.
>
>
> Key: HDDS-2098
> URL: https://issues.apache.org/jira/browse/HDDS-2098
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Affects Versions: 0.5.0
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> *Exception Trace*
> {code}
> log4j:ERROR Could not read configuration file from URL
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> java.io.FileNotFoundException: /etc/ozone/conf/ozone-shell-log4j.properties
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.(FileInputStream.java:138)
> at java.io.FileInputStream.(FileInputStream.java:93)
> at
> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
> at
> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
> at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66)
> at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
> at
> org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
> at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
> at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
> at org.apache.hadoop.ozone.web.ozShell.Shell.(Shell.java:35)
> log4j:ERROR Ignoring configuration file
> [file:/etc/ozone/conf/ozone-shell-log4j.properties].
> log4j:WARN No appenders could be found for logger
> (io.jaegertracing.thrift.internal.senders.ThriftSenderFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> {
> "metadata" : { },
> "name" : "vol-test-putfile-1567740142",
> "admin" : "root",
> "owner" : "root",
> "creationTime" : 1567740146501,
> "acls" : [ {
> "type" : "USER",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "root",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ],
> "quota" : 1152921504606846976
> }
> {code}
> *Fix*
> When a log4j file is not present, the default should be console.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2027) Corroborate log purging in TestOzoneManagerHA#testOMRestart
[ https://issues.apache.org/jira/browse/HDDS-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-2027: --- Assignee: Aravindan Vijayan > Corroborate log purging in TestOzoneManagerHA#testOMRestart > --- > > Key: HDDS-2027 > URL: https://issues.apache.org/jira/browse/HDDS-2027 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: test >Reporter: Hanisha Koneru >Assignee: Aravindan Vijayan >Priority: Major > > TestOzoneManagerHA#testOMRestart verifies that on OM restart, the snapshot is > downloaded and the state is reloaded properly. For this, it is assumed that > the logs are being purged periodically on the OMs. This Jira aims to > corroborate that the ratis log purging is happening as expected. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1054) List Multipart uploads in a bucket
[ https://issues.apache.org/jira/browse/HDDS-1054?focusedWorklogId=313270=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313270 ] ASF GitHub Bot logged work on HDDS-1054: Author: ASF GitHub Bot Created on: 16/Sep/19 20:04 Start Date: 16/Sep/19 20:04 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #1277: HDDS-1054. List Multipart uploads in a bucket URL: https://github.com/apache/hadoop/pull/1277#issuecomment-531935761 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313270) Time Spent: 9.5h (was: 9h 20m) > List Multipart uploads in a bucket > -- > > Key: HDDS-1054 > URL: https://issues.apache.org/jira/browse/HDDS-1054 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Elek, Marton >Priority: Blocker > Labels: pull-request-available > Time Spent: 9.5h > Remaining Estimate: 0h > > This Jira is to implement in ozone to list of in-progress multipart uploads > in a bucket. > [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListMPUpload.html] > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1054) List Multipart uploads in a bucket
[
https://issues.apache.org/jira/browse/HDDS-1054?focusedWorklogId=313268=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313268
]
ASF GitHub Bot logged work on HDDS-1054:
Author: ASF GitHub Bot
Created on: 16/Sep/19 20:02
Start Date: 16/Sep/19 20:02
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1277:
HDDS-1054. List Multipart uploads in a bucket
URL: https://github.com/apache/hadoop/pull/1277#discussion_r324859826
##
File path:
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3StorageType.java
##
@@ -52,4 +52,12 @@ public static S3StorageType getDefault() {
return STANDARD;
}
+ public static S3StorageType fromReplicationType(
+ ReplicationType replicationType) {
+if (replicationType == ReplicationType.STAND_ALONE) {
+ return S3StorageType.REDUCED_REDUNDANCY;
Review comment:
just a question, I think this is before even this patch.
Previously we use STAND_ALONE for replication factor one, now we use RATIS
with one and three. So, I think we need to change the code for this right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313268)
Time Spent: 9h 20m (was: 9h 10m)
> List Multipart uploads in a bucket
> --
>
> Key: HDDS-1054
> URL: https://issues.apache.org/jira/browse/HDDS-1054
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> This Jira is to implement in ozone to list of in-progress multipart uploads
> in a bucket.
> [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListMPUpload.html]
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2107) Datanodes should retry forever to connect to SCM in an unsecure environment
[
https://issues.apache.org/jira/browse/HDDS-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930825#comment-16930825
]
Hudson commented on HDDS-2107:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17309 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17309/])
HDDS-2107. Datanodes should retry forever to connect to SCM in an
(hanishakoneru: rev 66bd1681f8eec7c637a5deb0348f7b7901db64df)
* (edit)
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBufferWithOMResponse.java
* (edit)
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/SCMConnectionManager.java
> Datanodes should retry forever to connect to SCM in an unsecure environment
> ---
>
> Key: HDDS-2107
> URL: https://issues.apache.org/jira/browse/HDDS-2107
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.4.1
>Reporter: Vivek Ratnavel Subramanian
>Assignee: Vivek Ratnavel Subramanian
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> In an unsecure environment, the datanodes try upto 10 times after waiting for
> 1000 milliseconds each time before throwing this error:
> {code:java}
> Unable to communicate to SCM server at scm:9861 for past 0 seconds.
> java.net.ConnectException: Call From scm/10.65.36.118 to scm:9861 failed on
> connection exception: java.net.ConnectException: Connection refused; For more
> details see: http://wiki.apache.org/hadoop/ConnectionRefused
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1515)
> at org.apache.hadoop.ipc.Client.call(Client.java:1457)
> at org.apache.hadoop.ipc.Client.call(Client.java:1367)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy33.getVersion(Unknown Source)
> at
> org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolClientSideTranslatorPB.getVersion(StorageContainerDatanodeProtocolClientSideTranslatorPB.java:112)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:70)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:42)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:690)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:794)
> at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:411)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1572)
> at org.apache.hadoop.ipc.Client.call(Client.java:1403)
> ... 13 more
> {code}
> The datanodes should try forever to connect with SCM and not throw any errors.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1879) Support multiple excluded scopes when choosing datanodes in NetworkTopology
[ https://issues.apache.org/jira/browse/HDDS-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDDS-1879: - Fix Version/s: 0.5.0 Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~Sammi] for the contribution. I've merged the change to trunk. > Support multiple excluded scopes when choosing datanodes in NetworkTopology > --- > > Key: HDDS-1879 > URL: https://issues.apache.org/jira/browse/HDDS-1879 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2107) Datanodes should retry forever to connect to SCM in an unsecure environment
[
https://issues.apache.org/jira/browse/HDDS-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru resolved HDDS-2107.
--
Resolution: Fixed
> Datanodes should retry forever to connect to SCM in an unsecure environment
> ---
>
> Key: HDDS-2107
> URL: https://issues.apache.org/jira/browse/HDDS-2107
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.4.1
>Reporter: Vivek Ratnavel Subramanian
>Assignee: Vivek Ratnavel Subramanian
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> In an unsecure environment, the datanodes try upto 10 times after waiting for
> 1000 milliseconds each time before throwing this error:
> {code:java}
> Unable to communicate to SCM server at scm:9861 for past 0 seconds.
> java.net.ConnectException: Call From scm/10.65.36.118 to scm:9861 failed on
> connection exception: java.net.ConnectException: Connection refused; For more
> details see: http://wiki.apache.org/hadoop/ConnectionRefused
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1515)
> at org.apache.hadoop.ipc.Client.call(Client.java:1457)
> at org.apache.hadoop.ipc.Client.call(Client.java:1367)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy33.getVersion(Unknown Source)
> at
> org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolClientSideTranslatorPB.getVersion(StorageContainerDatanodeProtocolClientSideTranslatorPB.java:112)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:70)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:42)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:690)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:794)
> at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:411)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1572)
> at org.apache.hadoop.ipc.Client.call(Client.java:1403)
> ... 13 more
> {code}
> The datanodes should try forever to connect with SCM and not throw any errors.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2107) Datanodes should retry forever to connect to SCM in an unsecure environment
[
https://issues.apache.org/jira/browse/HDDS-2107?focusedWorklogId=313264=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313264
]
ASF GitHub Bot logged work on HDDS-2107:
Author: ASF GitHub Bot
Created on: 16/Sep/19 19:58
Start Date: 16/Sep/19 19:58
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on issue #1424: HDDS-2107.
Datanodes should retry forever to connect to SCM in an…
URL: https://github.com/apache/hadoop/pull/1424#issuecomment-531933488
Thank you @vivekratnavel. +1. I will commit it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313264)
Time Spent: 1h 20m (was: 1h 10m)
> Datanodes should retry forever to connect to SCM in an unsecure environment
> ---
>
> Key: HDDS-2107
> URL: https://issues.apache.org/jira/browse/HDDS-2107
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.4.1
>Reporter: Vivek Ratnavel Subramanian
>Assignee: Vivek Ratnavel Subramanian
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In an unsecure environment, the datanodes try upto 10 times after waiting for
> 1000 milliseconds each time before throwing this error:
> {code:java}
> Unable to communicate to SCM server at scm:9861 for past 0 seconds.
> java.net.ConnectException: Call From scm/10.65.36.118 to scm:9861 failed on
> connection exception: java.net.ConnectException: Connection refused; For more
> details see: http://wiki.apache.org/hadoop/ConnectionRefused
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1515)
> at org.apache.hadoop.ipc.Client.call(Client.java:1457)
> at org.apache.hadoop.ipc.Client.call(Client.java:1367)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy33.getVersion(Unknown Source)
> at
> org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolClientSideTranslatorPB.getVersion(StorageContainerDatanodeProtocolClientSideTranslatorPB.java:112)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:70)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:42)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:690)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:794)
> at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:411)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1572)
> at org.apache.hadoop.ipc.Client.call(Client.java:1403)
> ... 13 more
> {code}
> The datanodes should try forever to connect with SCM and not throw any errors.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HDDS-1879) Support multiple excluded scopes when choosing datanodes in NetworkTopology
[ https://issues.apache.org/jira/browse/HDDS-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930823#comment-16930823 ] Hudson commented on HDDS-1879: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17308 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17308/]) HDDS-1879. Support multiple excluded scopes when choosing datanodes in (xyao: rev 4222b62f2b257c6834122ff609c51fb7d5ab70c5) * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/InnerNodeImpl.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementRackAware.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetworkTopology.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetworkTopologyImpl.java * (edit) hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNetworkTopologyImpl.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementMetrics.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/InnerNode.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NetUtils.java * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementRackAware.java > Support multiple excluded scopes when choosing datanodes in NetworkTopology > --- > > Key: HDDS-1879 > URL: https://issues.apache.org/jira/browse/HDDS-1879 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Labels: pull-request-available > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2107) Datanodes should retry forever to connect to SCM in an unsecure environment
[
https://issues.apache.org/jira/browse/HDDS-2107?focusedWorklogId=313266=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313266
]
ASF GitHub Bot logged work on HDDS-2107:
Author: ASF GitHub Bot
Created on: 16/Sep/19 19:58
Start Date: 16/Sep/19 19:58
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #1424:
HDDS-2107. Datanodes should retry forever to connect to SCM in an…
URL: https://github.com/apache/hadoop/pull/1424
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313266)
Time Spent: 1.5h (was: 1h 20m)
> Datanodes should retry forever to connect to SCM in an unsecure environment
> ---
>
> Key: HDDS-2107
> URL: https://issues.apache.org/jira/browse/HDDS-2107
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.4.1
>Reporter: Vivek Ratnavel Subramanian
>Assignee: Vivek Ratnavel Subramanian
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> In an unsecure environment, the datanodes try upto 10 times after waiting for
> 1000 milliseconds each time before throwing this error:
> {code:java}
> Unable to communicate to SCM server at scm:9861 for past 0 seconds.
> java.net.ConnectException: Call From scm/10.65.36.118 to scm:9861 failed on
> connection exception: java.net.ConnectException: Connection refused; For more
> details see: http://wiki.apache.org/hadoop/ConnectionRefused
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1515)
> at org.apache.hadoop.ipc.Client.call(Client.java:1457)
> at org.apache.hadoop.ipc.Client.call(Client.java:1367)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy33.getVersion(Unknown Source)
> at
> org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolClientSideTranslatorPB.getVersion(StorageContainerDatanodeProtocolClientSideTranslatorPB.java:112)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:70)
> at
> org.apache.hadoop.ozone.container.common.states.endpoint.VersionEndpointTask.call(VersionEndpointTask.java:42)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:690)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:794)
> at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:411)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1572)
> at org.apache.hadoop.ipc.Client.call(Client.java:1403)
> ... 13 more
> {code}
> The datanodes should try forever to connect with SCM and not throw any errors.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To
[jira] [Work logged] (HDDS-1879) Support multiple excluded scopes when choosing datanodes in NetworkTopology
[ https://issues.apache.org/jira/browse/HDDS-1879?focusedWorklogId=313265=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313265 ] ASF GitHub Bot logged work on HDDS-1879: Author: ASF GitHub Bot Created on: 16/Sep/19 19:58 Start Date: 16/Sep/19 19:58 Worklog Time Spent: 10m Work Description: xiaoyuyao commented on issue #1194: HDDS-1879. Support multiple excluded scopes when choosing datanodes in NetworkTopology URL: https://github.com/apache/hadoop/pull/1194#issuecomment-531933511 +1, I've merged the change to trunk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313265) Time Spent: 4h 50m (was: 4h 40m) > Support multiple excluded scopes when choosing datanodes in NetworkTopology > --- > > Key: HDDS-1879 > URL: https://issues.apache.org/jira/browse/HDDS-1879 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Labels: pull-request-available > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1879) Support multiple excluded scopes when choosing datanodes in NetworkTopology
[ https://issues.apache.org/jira/browse/HDDS-1879?focusedWorklogId=313263=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313263 ] ASF GitHub Bot logged work on HDDS-1879: Author: ASF GitHub Bot Created on: 16/Sep/19 19:57 Start Date: 16/Sep/19 19:57 Worklog Time Spent: 10m Work Description: xiaoyuyao commented on pull request #1194: HDDS-1879. Support multiple excluded scopes when choosing datanodes in NetworkTopology URL: https://github.com/apache/hadoop/pull/1194 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313263) Time Spent: 4h 40m (was: 4.5h) > Support multiple excluded scopes when choosing datanodes in NetworkTopology > --- > > Key: HDDS-1879 > URL: https://issues.apache.org/jira/browse/HDDS-1879 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Labels: pull-request-available > Time Spent: 4h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1054) List Multipart uploads in a bucket
[
https://issues.apache.org/jira/browse/HDDS-1054?focusedWorklogId=313259=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313259
]
ASF GitHub Bot logged work on HDDS-1054:
Author: ASF GitHub Bot
Created on: 16/Sep/19 19:46
Start Date: 16/Sep/19 19:46
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1277:
HDDS-1054. List Multipart uploads in a bucket
URL: https://github.com/apache/hadoop/pull/1277#discussion_r324853402
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartUploadList.java
##
@@ -18,15 +18,21 @@
package org.apache.hadoop.ozone.om.helpers;
-import java.util.ArrayList;
import java.util.List;
+import org.apache.hadoop.hdds.client.ReplicationType;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor;
+
/**
- * List of in-flight MU upoads.
+ * List of in-flight MPU uploads.
*/
public class OmMultipartUploadList {
- private List uploads = new ArrayList<>();
+ private ReplicationType replicationType;
+
+ private ReplicationFactor replicationFactor;
Review comment:
Why do we need this in the multipart list result?
These are specific to each in-progress MPU.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313259)
Time Spent: 9h 10m (was: 9h)
> List Multipart uploads in a bucket
> --
>
> Key: HDDS-1054
> URL: https://issues.apache.org/jira/browse/HDDS-1054
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Elek, Marton
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 9h 10m
> Remaining Estimate: 0h
>
> This Jira is to implement in ozone to list of in-progress multipart uploads
> in a bucket.
> [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListMPUpload.html]
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2131) Optimize replication type and creation time calculation in S3 MPU list call
[
https://issues.apache.org/jira/browse/HDDS-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930808#comment-16930808
]
Bharat Viswanadham commented on HDDS-2131:
--
Hi [~swagle]
Thank You for taking up this jira.
This need to be fixed once HDDS-1054 got checked in.
> Optimize replication type and creation time calculation in S3 MPU list call
> ---
>
> Key: HDDS-2131
> URL: https://issues.apache.org/jira/browse/HDDS-2131
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Elek, Marton
>Assignee: Siddharth Wagle
>Priority: Major
>
> Based on the review from [~bharatviswa]:
> {code}
>
> hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
> metadataManager.getOpenKeyTable();
> OmKeyInfo omKeyInfo =
> openKeyTable.get(upload.getDbKey());
> {code}
> {quote}Here we are reading openKeyTable only for getting creation time. If we
> can have this information in omMultipartKeyInfo, we could avoid DB calls for
> openKeyTable.
> To do this, We can set creationTime in OmMultipartKeyInfo during
> initiateMultipartUpload . In this way, we can get all the required
> information from the MultipartKeyInfo table.
> And also StorageClass is missing from the returned OmMultipartUpload, as
> listMultipartUploads shows StorageClass information. For this, if we can
> return replicationType and depending on this value, we can set StorageClass
> in the listMultipartUploads Response.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2030) Generate simplifed reports by the dev-support/checks/*.sh scripts
[ https://issues.apache.org/jira/browse/HDDS-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-2030: --- Fix Version/s: 0.5.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Generate simplifed reports by the dev-support/checks/*.sh scripts > - > > Key: HDDS-2030 > URL: https://issues.apache.org/jira/browse/HDDS-2030 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 16.5h > Remaining Estimate: 0h > > hadoop-ozone/dev-support/checks directory contains shell scripts to execute > different type of code checks (findbugs, checkstyle, etc.) > Currently the contract is very simple. Every shell script executes one (and > only one) check and the shell response code is set according to the result > (non-zero code if failed). > To have better reporting in the github pr build, it would be great to improve > the scripts to generate simple summary files and save the relevant files for > archiving. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2030) Generate simplifed reports by the dev-support/checks/*.sh scripts
[ https://issues.apache.org/jira/browse/HDDS-2030?focusedWorklogId=313254=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313254 ] ASF GitHub Bot logged work on HDDS-2030: Author: ASF GitHub Bot Created on: 16/Sep/19 19:30 Start Date: 16/Sep/19 19:30 Worklog Time Spent: 10m Work Description: anuengineer commented on pull request #1348: HDDS-2030. Generate simplifed reports by the dev-support/checks/*.sh scripts URL: https://github.com/apache/hadoop/pull/1348 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313254) Time Spent: 16.5h (was: 16h 20m) > Generate simplifed reports by the dev-support/checks/*.sh scripts > - > > Key: HDDS-2030 > URL: https://issues.apache.org/jira/browse/HDDS-2030 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 16.5h > Remaining Estimate: 0h > > hadoop-ozone/dev-support/checks directory contains shell scripts to execute > different type of code checks (findbugs, checkstyle, etc.) > Currently the contract is very simple. Every shell script executes one (and > only one) check and the shell response code is set according to the result > (non-zero code if failed). > To have better reporting in the github pr build, it would be great to improve > the scripts to generate simple summary files and save the relevant files for > archiving. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2030) Generate simplifed reports by the dev-support/checks/*.sh scripts
[ https://issues.apache.org/jira/browse/HDDS-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930805#comment-16930805 ] Hudson commented on HDDS-2030: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17307 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17307/]) HDDS-2030. Generate simplifed reports by the dev-support/checks/*.sh (aengineer: rev c8d61ffef66ebc1d1ce2324787d0f1164d61e53f) * (edit) hadoop-ozone/dev-support/checks/findbugs.sh * (add) hadoop-ozone/dev-support/checks/_mvn_unit_report.sh * (edit) hadoop-ozone/dev-support/checks/acceptance.sh * (edit) hadoop-ozone/dev-support/checks/author.sh * (add) hadoop-ozone/dev-support/checks/README.md * (edit) hadoop-ozone/dev-support/checks/shellcheck.sh * (edit) hadoop-ozone/dev-support/checks/unit.sh * (edit) hadoop-ozone/dev-support/checks/integration.sh * (edit) hadoop-ozone/dev-support/checks/checkstyle.sh * (edit) hadoop-ozone/dev-support/checks/rat.sh > Generate simplifed reports by the dev-support/checks/*.sh scripts > - > > Key: HDDS-2030 > URL: https://issues.apache.org/jira/browse/HDDS-2030 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 16h 10m > Remaining Estimate: 0h > > hadoop-ozone/dev-support/checks directory contains shell scripts to execute > different type of code checks (findbugs, checkstyle, etc.) > Currently the contract is very simple. Every shell script executes one (and > only one) check and the shell response code is set according to the result > (non-zero code if failed). > To have better reporting in the github pr build, it would be great to improve > the scripts to generate simple summary files and save the relevant files for > archiving. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2030) Generate simplifed reports by the dev-support/checks/*.sh scripts
[ https://issues.apache.org/jira/browse/HDDS-2030?focusedWorklogId=313253=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313253 ] ASF GitHub Bot logged work on HDDS-2030: Author: ASF GitHub Bot Created on: 16/Sep/19 19:30 Start Date: 16/Sep/19 19:30 Worklog Time Spent: 10m Work Description: anuengineer commented on issue #1348: HDDS-2030. Generate simplifed reports by the dev-support/checks/*.sh scripts URL: https://github.com/apache/hadoop/pull/1348#issuecomment-531923640 +1, committed to the trunk. Thanks for the contributions, @elek and @adoroszlai This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313253) Time Spent: 16h 20m (was: 16h 10m) > Generate simplifed reports by the dev-support/checks/*.sh scripts > - > > Key: HDDS-2030 > URL: https://issues.apache.org/jira/browse/HDDS-2030 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 16h 20m > Remaining Estimate: 0h > > hadoop-ozone/dev-support/checks directory contains shell scripts to execute > different type of code checks (findbugs, checkstyle, etc.) > Currently the contract is very simple. Every shell script executes one (and > only one) check and the shell response code is set according to the result > (non-zero code if failed). > To have better reporting in the github pr build, it would be great to improve > the scripts to generate simple summary files and save the relevant files for > archiving. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2137) HddsClientUtils and OzoneUtils have duplicate verifyResourceName()
[ https://issues.apache.org/jira/browse/HDDS-2137?focusedWorklogId=313255=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313255 ] ASF GitHub Bot logged work on HDDS-2137: Author: ASF GitHub Bot Created on: 16/Sep/19 19:30 Start Date: 16/Sep/19 19:30 Worklog Time Spent: 10m Work Description: virajjasani commented on issue #1455: HDDS-2137 : OzoneUtils to verify resourceName using HddsClientUtils URL: https://github.com/apache/hadoop/pull/1455#issuecomment-531923682 @elek @adoroszlai Could you please let me know if there is a way to build hadoop-ozone with hadoop-hdds client dependency? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313255) Time Spent: 40m (was: 0.5h) > HddsClientUtils and OzoneUtils have duplicate verifyResourceName() > -- > > Key: HDDS-2137 > URL: https://issues.apache.org/jira/browse/HDDS-2137 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.5.0 >Reporter: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 40m > Remaining Estimate: 0h > > HddsClientUtils and OzoneUtils can share the method to verify resource name > that verifies if the bucket/volume name is a valid DNS name. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2120) Remove hadoop classes from ozonefs-current jar
[ https://issues.apache.org/jira/browse/HDDS-2120?focusedWorklogId=313252=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313252 ] ASF GitHub Bot logged work on HDDS-2120: Author: ASF GitHub Bot Created on: 16/Sep/19 19:26 Start Date: 16/Sep/19 19:26 Worklog Time Spent: 10m Work Description: arp7 commented on issue #1434: HDDS-2120. Remove hadoop classes from ozonefs-current jar URL: https://github.com/apache/hadoop/pull/1434#issuecomment-531922140 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313252) Time Spent: 1h 20m (was: 1h 10m) > Remove hadoop classes from ozonefs-current jar > -- > > Key: HDDS-2120 > URL: https://issues.apache.org/jira/browse/HDDS-2120 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > We have two kind of ozone file system jars: current and legacy. current is > designed to work only with exactly the same hadoop version which is used for > compilation (3.2 as of now). > But as of now the hadoop classes are included in the current jar which is not > necessary as the jar is expected to be used in an environment where the > hadoop classes (exactly the same hadoop classes) are already there. They can > be excluded. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2120) Remove hadoop classes from ozonefs-current jar
[ https://issues.apache.org/jira/browse/HDDS-2120?focusedWorklogId=313251=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313251 ] ASF GitHub Bot logged work on HDDS-2120: Author: ASF GitHub Bot Created on: 16/Sep/19 19:24 Start Date: 16/Sep/19 19:24 Worklog Time Spent: 10m Work Description: avijayanhwx commented on issue #1434: HDDS-2120. Remove hadoop classes from ozonefs-current jar URL: https://github.com/apache/hadoop/pull/1434#issuecomment-531921555 LGTM +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313251) Time Spent: 1h 10m (was: 1h) > Remove hadoop classes from ozonefs-current jar > -- > > Key: HDDS-2120 > URL: https://issues.apache.org/jira/browse/HDDS-2120 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > We have two kind of ozone file system jars: current and legacy. current is > designed to work only with exactly the same hadoop version which is used for > compilation (3.2 as of now). > But as of now the hadoop classes are included in the current jar which is not > necessary as the jar is expected to be used in an environment where the > hadoop classes (exactly the same hadoop classes) are already there. They can > be excluded. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2137) HddsClientUtils and OzoneUtils have duplicate verifyResourceName()
[ https://issues.apache.org/jira/browse/HDDS-2137?focusedWorklogId=313249=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313249 ] ASF GitHub Bot logged work on HDDS-2137: Author: ASF GitHub Bot Created on: 16/Sep/19 19:07 Start Date: 16/Sep/19 19:07 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1455: HDDS-2137 : OzoneUtils to verify resourceName using HddsClientUtils URL: https://github.com/apache/hadoop/pull/1455#issuecomment-531915276 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1184 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 72 | Maven dependency ordering for branch | | -1 | mvninstall | 31 | hadoop-ozone in trunk failed. | | -1 | compile | 22 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 62 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 927 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 168 | trunk passed | | 0 | spotbugs | 204 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 26 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for patch | | -1 | mvninstall | 34 | hadoop-ozone in the patch failed. | | -1 | compile | 25 | hadoop-ozone in the patch failed. | | -1 | javac | 25 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 28 | hadoop-hdds: The patch generated 7 new + 38 unchanged - 5 fixed = 45 total (was 43) | | -0 | checkstyle | 28 | hadoop-ozone: The patch generated 3 new + 26 unchanged - 9 fixed = 29 total (was 35) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 723 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 166 | the patch passed | | -1 | findbugs | 25 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 160 | hadoop-hdds in the patch failed. | | -1 | unit | 27 | hadoop-ozone in the patch failed. | | +1 | asflicense | 31 | The patch does not generate ASF License warnings. | | | | 4616 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.keyvalue.TestKeyValueContainer | | | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1455/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1455 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux eee188862a63 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 56f042c | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1455/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1455/1/artifact/out/branch-compile-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1455/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1455/1/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1455/1/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1455/1/artifact/out/patch-compile-hadoop-ozone.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1455/1/artifact/out/diff-checkstyle-hadoop-hdds.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1455/1/artifact/out/diff-checkstyle-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1455/1/artifact/out/patch-findbugs-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1455/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit |
[jira] [Work logged] (HDDS-2121) Create a shaded ozone filesystem (client) jar
[ https://issues.apache.org/jira/browse/HDDS-2121?focusedWorklogId=313230=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313230 ] ASF GitHub Bot logged work on HDDS-2121: Author: ASF GitHub Bot Created on: 16/Sep/19 18:13 Start Date: 16/Sep/19 18:13 Worklog Time Spent: 10m Work Description: arp7 commented on issue #1452: HDDS-2121. Create a shaded ozone filesystem (client) jar URL: https://github.com/apache/hadoop/pull/1452#issuecomment-531893729 The build seems to be failing in Jenkins. However I am able to compile Ozone locally with your patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313230) Time Spent: 40m (was: 0.5h) > Create a shaded ozone filesystem (client) jar > - > > Key: HDDS-2121 > URL: https://issues.apache.org/jira/browse/HDDS-2121 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Arpit Agarwal >Assignee: Elek, Marton >Priority: Blocker > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > We need a shaded Ozonefs jar that does not include Hadoop ecosystem > components (Hadoop, HDFS, Ratis, Zookeeper). > A common expected use case for Ozone is Hadoop clients (3.2.0 and later) > wanting to access Ozone via the Ozone Filesystem interface. For these > clients, we want to add Ozone file system jar to the classpath, however we > want to use Hadoop ecosystem dependencies that are `provided` and already > expected to be in the client classpath. > Note that this is different from the legacy jar which bundles a shaded Hadoop > 3.2.0. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14836) FileIoProvider should not increase FileIoErrors metric in datanode volume metric
[ https://issues.apache.org/jira/browse/HDFS-14836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-14836: --- Status: Patch Available (was: Open) LGTM +1 pending Jenkins > FileIoProvider should not increase FileIoErrors metric in datanode volume > metric > > > Key: HDFS-14836 > URL: https://issues.apache.org/jira/browse/HDFS-14836 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Aiphago >Assignee: Aiphago >Priority: Minor > Attachments: HDFS-14836-trunk-001.patch, HDFS-14836.patch > > > I found that FileIoErrors metric will increase in > BlockSender.sendPacket(),when use fileIoProvider.transferToSocketFully().But > in https://issues.apache.org/jira/browse/HDFS-2054 the Exception has been > ignore like "Broken pipe" and "Connection reset" . > So should do a filter when fileIoProvider increase FileIoErrors count ? -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2111) XSS fragments can be injected to the S3g landing page
[ https://issues.apache.org/jira/browse/HDDS-2111?focusedWorklogId=313225=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313225 ] ASF GitHub Bot logged work on HDDS-2111: Author: ASF GitHub Bot Created on: 16/Sep/19 18:07 Start Date: 16/Sep/19 18:07 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1447: HDDS-2111. XSS fragments can be injected to the S3g landing page URL: https://github.com/apache/hadoop/pull/1447#issuecomment-531891282 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 43 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | -1 | mvninstall | 33 | hadoop-ozone in trunk failed. | | -1 | compile | 22 | hadoop-ozone in trunk failed. | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 879 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 169 | trunk passed | ||| _ Patch Compile Tests _ | | -1 | mvninstall | 33 | hadoop-ozone in the patch failed. | | -1 | jshint | 83 | The patch generated 1392 new + 2737 unchanged - 0 fixed = 4129 total (was 2737) | | -1 | compile | 24 | hadoop-ozone in the patch failed. | | -1 | javac | 24 | hadoop-ozone in the patch failed. | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 677 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 156 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 189 | hadoop-hdds in the patch failed. | | -1 | unit | 25 | hadoop-ozone in the patch failed. | | +1 | asflicense | 31 | The patch does not generate ASF License warnings. | | | | 2723 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.keyvalue.TestKeyValueContainer | | | hadoop.ozone.container.common.TestDatanodeStateMachine | | | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1447 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient jshint | | uname | Linux e3e3320f7474 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 56f042c | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/2/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/2/artifact/out/branch-compile-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/2/artifact/out/patch-mvninstall-hadoop-ozone.txt | | jshint | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/2/artifact/out/diff-patch-jshint.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/2/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/2/artifact/out/patch-compile-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/2/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/2/testReport/ | | Max. process+thread count | 413 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/s3gateway U: hadoop-ozone/s3gateway | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1447/2/console | | versions | git=2.7.4 maven=3.3.9 jshint=2.10.2 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this
[jira] [Work started] (HDDS-2138) OM bucket operations do not add up
[ https://issues.apache.org/jira/browse/HDDS-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDDS-2138 started by Doroszlai, Attila. --- > OM bucket operations do not add up > -- > > Key: HDDS-2138 > URL: https://issues.apache.org/jira/browse/HDDS-2138 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.4.1 >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Minor > Attachments: bucket.png > > > Total OM bucket operations may be higher than sum of counts for individual > operation type, because S3 bucket operations are displayed in separate charts. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2138) OM bucket operations do not add up
Doroszlai, Attila created HDDS-2138: --- Summary: OM bucket operations do not add up Key: HDDS-2138 URL: https://issues.apache.org/jira/browse/HDDS-2138 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Manager Affects Versions: 0.4.1 Reporter: Doroszlai, Attila Assignee: Doroszlai, Attila Attachments: bucket.png Total OM bucket operations may be higher than sum of counts for individual operation type, because S3 bucket operations are displayed in separate charts. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2138) OM bucket operations do not add up
[ https://issues.apache.org/jira/browse/HDDS-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doroszlai, Attila updated HDDS-2138: Attachment: bucket.png > OM bucket operations do not add up > -- > > Key: HDDS-2138 > URL: https://issues.apache.org/jira/browse/HDDS-2138 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Affects Versions: 0.4.1 >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Minor > Attachments: bucket.png > > > Total OM bucket operations may be higher than sum of counts for individual > operation type, because S3 bucket operations are displayed in separate charts. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
