[jira] [Assigned] (HDDS-1437) TestBlockOutputStreamWithFailures#testWatchForCommitDatanodeFailure fails with assertion error
[
https://issues.apache.org/jira/browse/HDDS-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee reassigned HDDS-1437:
-
Assignee: Shashikant Banerjee
> TestBlockOutputStreamWithFailures#testWatchForCommitDatanodeFailure fails
> with assertion error
> --
>
> Key: HDDS-1437
> URL: https://issues.apache.org/jira/browse/HDDS-1437
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Major
>
> The test is failing with the following assertion
> {code}
> java.lang.AssertionError: expected:<2> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures.testWatchForCommitDatanodeFailure(TestBlockOutputStreamWithFailures.java:373)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
> https://ci.anzix.net//job/ozone-nightly/62//testReport/junit/org.apache.hadoop.ozone.client.rpc/TestBlockOutputStreamWithFailures/testWatchForCommitDatanodeFailure/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14476) lock too long when fix inconsistent blocks between disk and in-memory
Sean Chow created HDFS-14476: Summary: lock too long when fix inconsistent blocks between disk and in-memory Key: HDFS-14476 URL: https://issues.apache.org/jira/browse/HDFS-14476 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 2.7.0, 2.6.0 Reporter: Sean Chow When directoryScanner have the results of differences between disk and in-memory blocks. it will try to run `checkAndUpdate` to fix it. However `FsDatasetImpl.checkAndUpdate` is a synchronized call As I have about 6millions blocks for every datanodes and every 6hours' scan will have about 25000 abnormal blocks to fix. That leads to a long lock holding FsDatasetImpl object. let's assume every block need 10ms to fix(because of latency of SAS disk), that will cost 250 seconds to finish. That means all reads and writes will be blocked for 3mins for that datanode. ``` 2019-05-06 08:06:51,704 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool BP-1644920766-10.223.143.220-1450099987967 Total blocks: 6850197, missing metadata files:23574, missing block files:23574, missing blocks in memory:47625, mismatched blocks:0 ... 2019-05-06 08:16:41,625 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Took 588402ms to process 1 commands from NN ``` Take long time to process command from nn because threads are blocked. And namenode will see long lastContact time for this datanode. Maybe this affect all hdfs versions. to fix: just like process invalidate command from namenode with 1000 batch size, fix these abnormal block should be handled with batch too and sleep 2 seconds between the batch to allow normal reading/writing blocks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1451) SCMBlockManager findPipeline and createPipeline are not lock protected
[
https://issues.apache.org/jira/browse/HDDS-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834389#comment-16834389
]
Mukul Kumar Singh commented on HDDS-1451:
-
[~avijayan], yes you are absolutely correct. The problem is exactly the one you
have described. Some pipelines can be created between the point of check for
pipeline and point of allocating the pipeline and then the pipeline creation
will fail
> SCMBlockManager findPipeline and createPipeline are not lock protected
> --
>
> Key: HDDS-1451
> URL: https://issues.apache.org/jira/browse/HDDS-1451
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: MiniOzoneChaosCluster
>
> SCM BlockManager may try to allocate pipelines in the cases when it is not
> needed. This happens because BlockManagerImpl#allocateBlock is not lock
> protected, so multiple pipelines can be allocated from it. One of the
> pipeline allocation can fail even when one of the existing pipeline already
> exists.
> {code}
> 2019-04-22 22:34:14,336 INFO pipeline.RatisPipelineProvider
> (RatisPipelineProvider.java:lambda$create$1(103)) - pipeline Pipeline[ Id:
> 6f4bb2d7-d660-4f9f-bc06-72b10f9a738e, Nodes: 76e1a493-fd55-4d67-9f5
> 5-c04fd6bd3a33{ip: 192.168.0.104, host: 192.168.0.104, certSerialId:
> null}2b9850b2-aed3-4a40-91b5-2447dc5246bf{ip: 192.168.0.104, host:
> 192.168.0.104, certSerialId: null}12248721-ea6a-453f-8dad-fc7fbe692f
> d2{ip: 192.168.0.104, host: 192.168.0.104, certSerialId: null}, Type:RATIS,
> Factor:THREE, State:OPEN]
> 2019-04-22 22:34:14,386 INFO impl.RoleInfo
> (RoleInfo.java:shutdownLeaderElection(134)) -
> e17b7852-4691-40c7-8791-ad0b0da5201f: shutdown LeaderElection
> 2019-04-22 22:34:14,388 INFO pipeline.RatisPipelineProvider
> (RatisPipelineProvider.java:lambda$create$1(103)) - pipeline Pipeline[ Id:
> 552e28f3-98d9-41f3-86e0-c1b9494838a5, Nodes: e17b7852-4691-40c7-879
> 1-ad0b0da5201f{ip: 192.168.0.104, host: 192.168.0.104, certSerialId:
> null}fd365bac-e26e-4b11-afd8-9d08cd1b0521{ip: 192.168.0.104, host:
> 192.168.0.104, certSerialId: null}9583a007-7f02-4074-9e26-19bc18e29e
> c5{ip: 192.168.0.104, host: 192.168.0.104, certSerialId: null}, Type:RATIS,
> Factor:THREE, State:OPEN]
> 2019-04-22 22:34:14,388 INFO impl.RoleInfo (RoleInfo.java:updateAndGet(143))
> - e17b7852-4691-40c7-8791-ad0b0da5201f: start FollowerState
> 2019-04-22 22:34:14,388 INFO pipeline.RatisPipelineProvider
> (RatisPipelineProvider.java:lambda$create$1(103)) - pipeline Pipeline[ Id:
> 5383151b-d625-4362-a7dd-c0d353acaf76, Nodes: 80f16ad6-3879-4a64-a3c
> 7-7719813cc139{ip: 192.168.0.104, host: 192.168.0.104, certSerialId:
> null}082ce481-7fb0-4f88-ac21-82609290a6a2{ip: 192.168.0.104, host:
> 192.168.0.104, certSerialId: null}dd5f5a70-0217-4577-b7a2-c42aa139d1
> 8a{ip: 192.168.0.104, host: 192.168.0.104, certSerialId: null}, Type:RATIS,
> Factor:THREE, State:OPEN]
> 2019-04-22 22:34:14,389 INFO pipeline.RatisPipelineProvider
> (RatisPipelineProvider.java:lambda$create$1(103)) - pipeline Pipeline[ Id:
> be4854e5-7933-4caa-b32e-f482cf500247, Nodes: 6e2356f1-479d-498b-876
> a-1c90623c498b{ip: 192.168.0.104, host: 192.168.0.104, certSerialId:
> null}8ac46d94-9975-4eea-9448-2618c69d7bf3{ip: 192.168.0.104, host:
> 192.168.0.104, certSerialId: null}a3ed36a1-44ca-47b2-b9b3-5aeef04595
> 18{ip: 192.168.0.104, host: 192.168.0.104, certSerialId: null}, Type:RATIS,
> Factor:THREE, State:OPEN]
> 2019-04-22 22:34:14,390 INFO pipeline.RatisPipelineProvider
> (RatisPipelineProvider.java:lambda$create$1(103)) - pipeline Pipeline[ Id:
> 21e368e2-f82a-4c61-9cc3-06e8de22ea6b, Nodes:
> 82632040-5754-4122-b187-331879586842{ip: 192.168.0.104, host: 192.168.0.104,
> certSerialId: null}923c8537-b869-4085-adcb-0a9accdcd089{ip: 192.168.0.104,
> host: 192.168.0.104, certSerialId:
> null}c6d790bf-e3a6-4064-acb5-f74796cd38a9{ip: 192.168.0.104, host:
> 192.168.0.104, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN]
> 2019-04-22 22:34:14,390 INFO pipeline.RatisPipelineProvider
> (RatisPipelineProvider.java:lambda$create$1(103)) - pipeline Pipeline[ Id:
> cccbc2ed-e0e2-4578-a8a2-94f4b645be52, Nodes:
> 91ae6848-a778-43be-a4a1-5855f7adc0d8{ip: 192.168.0.104, host: 192.168.0.104,
> certSerialId: null}8f330a03-40e2-4bd1-9b43-5e05b13d89f0{ip: 192.168.0.104,
> host: 192.168.0.104, certSerialId:
> null}4f3070dc-650b-48d7-87b5-d2076104e7b4{ip: 192.168.0.104, host:
> 192.168.0.104, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN]
> 2019-04-22 22:34:14,392 ERROR block.BlockManagerImpl
> (BlockManagerImpl.java:allocateBlock(192)) - Pipeline creation failed for
>
[jira] [Work logged] (HDDS-1461) Optimize listStatus api in OzoneFileSystem
[
https://issues.apache.org/jira/browse/HDDS-1461?focusedWorklogId=238254=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238254
]
ASF GitHub Bot logged work on HDDS-1461:
Author: ASF GitHub Bot
Created on: 07/May/19 05:02
Start Date: 07/May/19 05:02
Worklog Time Spent: 10m
Work Description: mukul1987 commented on pull request #782: HDDS-1461.
Optimize listStatus api in OzoneFileSystem
URL: https://github.com/apache/hadoop/pull/782#discussion_r281460619
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -1546,6 +1552,101 @@ public OmKeyInfo lookupFile(OmKeyArgs args) throws
IOException {
ResultCodes.NOT_A_FILE);
}
+ /**
+ * List the status for a file or a directory and its contents.
+ *
+ * @param args Key args
+ * @param recursive For a directory if true all the descendants of a
+ * particular directory are listed
+ * @param startKey Key from which listing needs to start. If startKey
exists
+ * its status is included in the final list.
+ * @param numEntries Number of entries to list from the start key
+ * @return list of file status
+ */
+ public List listStatus(OmKeyArgs args, boolean recursive,
+ String startKey, long numEntries) throws IOException {
+Preconditions.checkNotNull(args, "Key args can not be null");
+String volumeName = args.getVolumeName();
+String bucketName = args.getBucketName();
+String keyName = args.getKeyName();
+
+List fileStatusList = new ArrayList<>();
+try {
+ metadataManager.getLock().acquireBucketLock(volumeName, bucketName);
+ if (Strings.isNullOrEmpty(startKey)) {
+OzoneFileStatus fileStatus = getFileStatus(args);
+if (fileStatus.isFile()) {
+ return Collections.singletonList(fileStatus);
+}
+startKey = OzoneFSUtils.addTrailingSlashIfNeeded(keyName);
+ }
+
+ String seekKeyInDb =
+ metadataManager.getOzoneKey(volumeName, bucketName, startKey);
+ String keyInDb = OzoneFSUtils.addTrailingSlashIfNeeded(
+ metadataManager.getOzoneKey(volumeName, bucketName, keyName));
+ TableIterator>
+ iterator = metadataManager.getKeyTable().iterator();
+ iterator.seek(seekKeyInDb);
+
+ if (!iterator.hasNext()) {
+return Collections.emptyList();
+ }
+
+ if (iterator.key().equals(keyInDb)) {
+// skip the key which needs to be listed
+iterator.next();
+ }
+
+ while (iterator.hasNext() && numEntries - fileStatusList.size() > 0) {
+String entryInDb = iterator.key();
+OmKeyInfo value = iterator.value().getValue();
+if (entryInDb.startsWith(keyInDb)) {
+ String entryKeyName = value.getKeyName();
+ if (recursive) {
+// for recursive list all the entries
+fileStatusList.add(new OzoneFileStatus(value, scmBlockSize,
+!OzoneFSUtils.isFile(entryKeyName)));
+iterator.next();
+ } else {
+// get the child of the directory to list from the entry. For
+// example if directory to list is /a and entry is /a/b/c where
+// c is a file. The immediate child is b which is a directory. c
+// should not be listed as child of a.
+String immediateChild = OzoneFSUtils
+.getImmediateChild(entryKeyName, keyName);
+boolean isFile = OzoneFSUtils.isFile(immediateChild);
+if (isFile) {
+ fileStatusList
+ .add(new OzoneFileStatus(value, scmBlockSize, !isFile));
+ iterator.next();
+} else {
+ // if entry is a directory
+ fileStatusList.add(new OzoneFileStatus(immediateChild));
+ // skip the other descendants of this child directory.
+ iterator.seek(
+ getNextGreaterString(volumeName, bucketName,
immediateChild));
+}
+ }
+} else {
+ break;
+}
+ }
+} finally {
+ metadataManager.getLock().releaseBucketLock(volumeName, bucketName);
+}
+return fileStatusList;
+ }
+
+ private String getNextGreaterString(String volumeName, String bucketName,
+ String keyPrefix) {
+// TODO: Use string codec
+// Increment the last character of the string and return the new ozone key.
+String nextPrefix = keyPrefix.substring(0, keyPrefix.length() - 1) +
+String.valueOf((char) (keyPrefix.charAt(keyPrefix.length() - 1) + 1));
Review comment:
This is a great optimization.
Should this point to first character in the ASCII table ? Also lets verify
that this for UTF-8 encoding as well.
This is an
[jira] [Commented] (HDFS-13995) RBF: Security documentation
[ https://issues.apache.org/jira/browse/HDFS-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834381#comment-16834381 ] CR Hota commented on HDFS-13995: [~brahmareddy] [~ajisakaa] Thanks for the review. Am not very comfortable adding details of zookeeper operations in the router manual. We could rather add a link pointing to zookeeper apache homepage. jute maxbuffer is something that has evolved in meaning over different releases of zookeeper (client side vs server side etc). Depending on what versions users use, they would need to refer to that doc. The reason I just touched upon this property is that its a critical property so users at least know what to read/find in zookeeper. > RBF: Security documentation > --- > > Key: HDFS-13995 > URL: https://issues.apache.org/jira/browse/HDFS-13995 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-13995-HDFS-13891.001.patch, > HDFS-13995-HDFS-13891.002.patch, HDFS-13995-HDFS-13891.003.patch > > > Documentation for users under the section relating to security needs to be > updated once security work is completely. > [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html#Security] > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14358) Provide LiveNode and DeadNode filter in DataNode UI
[ https://issues.apache.org/jira/browse/HDFS-14358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina reassigned HDFS-14358: Assignee: hemanthboyina > Provide LiveNode and DeadNode filter in DataNode UI > --- > > Key: HDFS-14358 > URL: https://issues.apache.org/jira/browse/HDFS-14358 > Project: Hadoop HDFS > Issue Type: Wish >Affects Versions: 3.1.2 >Reporter: Ravuri Sushma sree >Assignee: hemanthboyina >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1395) Key write fails with BlockOutputStream has been closed exception
[
https://issues.apache.org/jira/browse/HDDS-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDDS-1395:
--
Status: Open (was: Patch Available)
> Key write fails with BlockOutputStream has been closed exception
>
>
> Key: HDDS-1395
> URL: https://issues.apache.org/jira/browse/HDDS-1395
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Attachments: HDDS-1395.000.patch, HDDS-1395.001.patch,
> HDDS-1395.003.patch
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Key write fails with BlockOutputStream has been closed
> {code}
> 2019-04-05 11:24:47,770 ERROR ozone.MiniOzoneLoadGenerator
> (MiniOzoneLoadGenerator.java:load(102)) - LOADGEN: Create
> key:pool-431-thread-9-2092651262 failed with exception, but skipping
> java.io.IOException: BlockOutputStream has been closed.
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.checkOpen(BlockOutputStream.java:662)
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:245)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:131)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:325)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:287)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49)
> at java.io.OutputStream.write(OutputStream.java:75)
> at
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.load(MiniOzoneLoadGenerator.java:100)
> at
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$0(MiniOzoneLoadGenerator.java:143)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1395) Key write fails with BlockOutputStream has been closed exception
[
https://issues.apache.org/jira/browse/HDDS-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDDS-1395:
--
Status: Patch Available (was: Open)
> Key write fails with BlockOutputStream has been closed exception
>
>
> Key: HDDS-1395
> URL: https://issues.apache.org/jira/browse/HDDS-1395
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Attachments: HDDS-1395.000.patch, HDDS-1395.001.patch,
> HDDS-1395.003.patch
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Key write fails with BlockOutputStream has been closed
> {code}
> 2019-04-05 11:24:47,770 ERROR ozone.MiniOzoneLoadGenerator
> (MiniOzoneLoadGenerator.java:load(102)) - LOADGEN: Create
> key:pool-431-thread-9-2092651262 failed with exception, but skipping
> java.io.IOException: BlockOutputStream has been closed.
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.checkOpen(BlockOutputStream.java:662)
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:245)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:131)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:325)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:287)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49)
> at java.io.OutputStream.write(OutputStream.java:75)
> at
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.load(MiniOzoneLoadGenerator.java:100)
> at
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$0(MiniOzoneLoadGenerator.java:143)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1491) Ozone KeyInputStream seek() should not read the chunk file
[ https://issues.apache.org/jira/browse/HDDS-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-1491: - Target Version/s: 0.5.0 > Ozone KeyInputStream seek() should not read the chunk file > -- > > Key: HDDS-1491 > URL: https://issues.apache.org/jira/browse/HDDS-1491 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > KeyInputStream#seek() calls BlockInputStream#seek() to adjust the buffer > position to the seeked position. As part of the seek operation, the whole > chunk is read from the container and stored in the buffer so that the buffer > position can be advanced to the seeked position. > We should not read from disk on a seek() operation. Instead, for a read > operation, when the chunk file is read and put in the buffer, at that time, > we can advance the buffer position to the previously seeked position. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1496) readChunkFromContainer() should only read the required part of chunk file
[ https://issues.apache.org/jira/browse/HDDS-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-1496: - Target Version/s: 0.5.0 > readChunkFromContainer() should only read the required part of chunk file > - > > Key: HDDS-1496 > URL: https://issues.apache.org/jira/browse/HDDS-1496 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > > BlockInputStream#readChunkFromContainer() reads the whole chunk from disk > even if we need to read only a part of the chunk. > This Jira aims to improve readChunkFromContainer so that only that part of > the chunk file is read which is needed by client plus the part of chunk file > which is required to verify the checksum. > For example, lets say the client is reading from index 120 to 450 in the > chunk. And let's say checksum is stored for every 100 bytes in the chunk i.e. > the first checksum is for bytes from index 0 to 99, the next for bytes from > index 100 to 199 and so on. To verify bytes from 120 to 450, we would need to > read from bytes 100 to 499 so that checksum verification can be done. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1496) readChunkFromContainer() should only read the required part of chunk file
Hanisha Koneru created HDDS-1496: Summary: readChunkFromContainer() should only read the required part of chunk file Key: HDDS-1496 URL: https://issues.apache.org/jira/browse/HDDS-1496 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Hanisha Koneru Assignee: Hanisha Koneru BlockInputStream#readChunkFromContainer() reads the whole chunk from disk even if we need to read only a part of the chunk. This Jira aims to improve readChunkFromContainer so that only that part of the chunk file is read which is needed by client plus the part of chunk file which is required to verify the checksum. For example, lets say the client is reading from index 120 to 450 in the chunk. And let's say checksum is stored for every 100 bytes in the chunk i.e. the first checksum is for bytes from index 0 to 99, the next for bytes from index 100 to 199 and so on. To verify bytes from 120 to 450, we would need to read from bytes 100 to 499 so that checksum verification can be done. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14469) FsDatasetImpl() throws a DiskErrorException when the configuration has wrong values, results in unnecessary retry
[
https://issues.apache.org/jira/browse/HDFS-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
eBugs updated HDFS-14469:
-
Summary: FsDatasetImpl() throws a DiskErrorException when the configuration
has wrong values, results in unnecessary retry (was: FsDatasetImpl() throws a
DiskErrorException when the configuration has wrong values)
> FsDatasetImpl() throws a DiskErrorException when the configuration has wrong
> values, results in unnecessary retry
> -
>
> Key: HDFS-14469
> URL: https://issues.apache.org/jira/browse/HDFS-14469
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: eBugs
>Priority: Minor
>
> Dear HDFS developers, we are developing a tool to detect exception-related
> bugs in Java. Our prototype has spotted the following {{throw}} statement
> whose exception class and error message indicate different error conditions.
>
> Version: Hadoop-3.1.2
> File:
> HADOOP-ROOT/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
> Line: 294-297
> {code:java}
> throw new DiskErrorException("Invalid value configured for "
> + "dfs.datanode.failed.volumes.tolerated - " + volFailuresTolerated
> + ". Value configured is either less than maxVolumeFailureLimit or
> greater than "
> + "to the number of configured volumes (" + volsConfigured + ").");{code}
>
> A {{DiskErrorException}} means an error has occurred when the process is
> interacting with the disk, e.g., in
> {{org.apache.hadoop.util.DiskChecker.checkDirInternal()}} we have the
> following code (lines 97-98):
> {code:java}
> throw new DiskErrorException("Cannot create directory: " +
> dir.toString());{code}
> However, the error message of the first exception indicates that
> {{dfs.datanode.failed.volumes.tolerated}} is configured incorrectly, which
> means there is nothing wrong with the disk (yet). This mismatch could be a
> problem. For example, the callers trying to handle other
> {{DiskErrorException}} may accidentally (and incorrectly) handle the
> configuration error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14469) FsDatasetImpl() throws a DiskErrorException when the configuration has wrong values
[
https://issues.apache.org/jira/browse/HDFS-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834358#comment-16834358
]
eBugs edited comment on HDFS-14469 at 5/7/19 3:59 AM:
--
After looking at the related code, we have found the following call stack:
{noformat}
BPServiceActor.run()
-> BPServiceActor.connectToNNAndHandshake()
-> BPOfferService.verifyAndSetNamespaceInfo()
-> DataNode.initBlockPool()
-> DataNode.initStorage()
-> FsDatasetFactory.newInstance()
-> FsDatasetImpl(){noformat}
If this call stack is valid, then IMHO we may have a minor issue:
Since {{BPServiceActor.run()}} retries all {{IOException}} from
{{BPServiceActor.connectToNNAndHandshake()}}, the {{DiskErrorException}} thrown
by {{FsDatasetImpl()}} will also result in a retry. However, since the
exception is thrown because of user's incorrect configuration, the retry here
is wasted.
was (Author: ebugs-in-cloud-systems):
After looking at the related code, we have found the following call stack:
{noformat}
BPServiceActor.run()
-> BPServiceActor.connectToNNAndHandshake()
-> BPOfferService.verifyAndSetNamespaceInfo()
-> DataNode.initBlockPool()
-> DataNode.initStorage()
-> FsDatasetFactory.newInstance()
-> FsDatasetImpl(){noformat}
If this call stack is valid, then IMHO we may have a minor issue:
Since {{BPServiceActor.run()}} retries all {{IOException}} from
{{BPServiceActor.connectToNNAndHandshake()}}, the {{DiskErrorException}} thrown
by {{FsDatasetImpl()}} will also be retried. However, since the exception is
thrown because of user's incorrect configuration, the retry here is wasted.
> FsDatasetImpl() throws a DiskErrorException when the configuration has wrong
> values
> ---
>
> Key: HDFS-14469
> URL: https://issues.apache.org/jira/browse/HDFS-14469
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: eBugs
>Priority: Minor
>
> Dear HDFS developers, we are developing a tool to detect exception-related
> bugs in Java. Our prototype has spotted the following {{throw}} statement
> whose exception class and error message indicate different error conditions.
>
> Version: Hadoop-3.1.2
> File:
> HADOOP-ROOT/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
> Line: 294-297
> {code:java}
> throw new DiskErrorException("Invalid value configured for "
> + "dfs.datanode.failed.volumes.tolerated - " + volFailuresTolerated
> + ". Value configured is either less than maxVolumeFailureLimit or
> greater than "
> + "to the number of configured volumes (" + volsConfigured + ").");{code}
>
> A {{DiskErrorException}} means an error has occurred when the process is
> interacting with the disk, e.g., in
> {{org.apache.hadoop.util.DiskChecker.checkDirInternal()}} we have the
> following code (lines 97-98):
> {code:java}
> throw new DiskErrorException("Cannot create directory: " +
> dir.toString());{code}
> However, the error message of the first exception indicates that
> {{dfs.datanode.failed.volumes.tolerated}} is configured incorrectly, which
> means there is nothing wrong with the disk (yet). This mismatch could be a
> problem. For example, the callers trying to handle other
> {{DiskErrorException}} may accidentally (and incorrectly) handle the
> configuration error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14469) FsDatasetImpl() throws a DiskErrorException when the configuration has wrong values
[
https://issues.apache.org/jira/browse/HDFS-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834358#comment-16834358
]
eBugs edited comment on HDFS-14469 at 5/7/19 3:58 AM:
--
After looking at the related code, we have found the following call stack:
{noformat}
BPServiceActor.run()
-> BPServiceActor.connectToNNAndHandshake()
-> BPOfferService.verifyAndSetNamespaceInfo()
-> DataNode.initBlockPool()
-> DataNode.initStorage()
-> FsDatasetFactory.newInstance()
-> FsDatasetImpl(){noformat}
If this call stack is valid, then IMHO we may have a minor issue:
Since {{BPServiceActor.run()}} retries all {{IOException}} from
{{BPServiceActor.connectToNNAndHandshake()}}, the {{DiskErrorException}} thrown
by {{FsDatasetImpl()}} will also be retried. However, since the exception is
thrown because of user's incorrect configuration, the retry here is wasted.
was (Author: ebugs-in-cloud-systems):
After looking at the related code, we have found the following call stack:
{noformat}
BPServiceActor.run()
-> BPServiceActor.connectToNNAndHandshake()
-> BPOfferService.verifyAndSetNamespaceInfo()
-> DataNode.initBlockPool()
-> DataNode.initStorage()
-> FsDatasetFactory.newInstance()
-> FsDatasetImpl(){noformat}
If this call stack is valid, then IMHO we may have a minor issue:
Since {{BPServiceActor.run()}} retries all {{IOException}} from
{{BPServiceActor.connectToNNAndHandshake()}}, the {{DiskErrorException}} thrown
by {{FsDatasetImpl()}} will also be retried. However, since the exception is
thrown because of user's incorrect configuration, retrying here will not help.
> FsDatasetImpl() throws a DiskErrorException when the configuration has wrong
> values
> ---
>
> Key: HDFS-14469
> URL: https://issues.apache.org/jira/browse/HDFS-14469
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: eBugs
>Priority: Minor
>
> Dear HDFS developers, we are developing a tool to detect exception-related
> bugs in Java. Our prototype has spotted the following {{throw}} statement
> whose exception class and error message indicate different error conditions.
>
> Version: Hadoop-3.1.2
> File:
> HADOOP-ROOT/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
> Line: 294-297
> {code:java}
> throw new DiskErrorException("Invalid value configured for "
> + "dfs.datanode.failed.volumes.tolerated - " + volFailuresTolerated
> + ". Value configured is either less than maxVolumeFailureLimit or
> greater than "
> + "to the number of configured volumes (" + volsConfigured + ").");{code}
>
> A {{DiskErrorException}} means an error has occurred when the process is
> interacting with the disk, e.g., in
> {{org.apache.hadoop.util.DiskChecker.checkDirInternal()}} we have the
> following code (lines 97-98):
> {code:java}
> throw new DiskErrorException("Cannot create directory: " +
> dir.toString());{code}
> However, the error message of the first exception indicates that
> {{dfs.datanode.failed.volumes.tolerated}} is configured incorrectly, which
> means there is nothing wrong with the disk (yet). This mismatch could be a
> problem. For example, the callers trying to handle other
> {{DiskErrorException}} may accidentally (and incorrectly) handle the
> configuration error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14469) FsDatasetImpl() throws a DiskErrorException when the configuration has wrong values
[
https://issues.apache.org/jira/browse/HDFS-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834358#comment-16834358
]
eBugs commented on HDFS-14469:
--
After looking at the related code, we have found the following call stack:
{noformat}
BPServiceActor.run()
-> BPServiceActor.connectToNNAndHandshake()
-> BPOfferService.verifyAndSetNamespaceInfo()
-> DataNode.initBlockPool()
-> DataNode.initStorage()
-> FsDatasetFactory.newInstance()
-> FsDatasetImpl(){noformat}
If this call stack is valid, then IMHO we may have a minor issue:
Since {{BPServiceActor.run()}} retries all {{IOException}} from
{{BPServiceActor.connectToNNAndHandshake()}}, the {{DiskErrorException}} thrown
by {{FsDatasetImpl()}} will also be retried. However, since the exception is
thrown because of user's incorrect configuration, retrying here will not help.
> FsDatasetImpl() throws a DiskErrorException when the configuration has wrong
> values
> ---
>
> Key: HDFS-14469
> URL: https://issues.apache.org/jira/browse/HDFS-14469
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: eBugs
>Priority: Minor
>
> Dear HDFS developers, we are developing a tool to detect exception-related
> bugs in Java. Our prototype has spotted the following {{throw}} statement
> whose exception class and error message indicate different error conditions.
>
> Version: Hadoop-3.1.2
> File:
> HADOOP-ROOT/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
> Line: 294-297
> {code:java}
> throw new DiskErrorException("Invalid value configured for "
> + "dfs.datanode.failed.volumes.tolerated - " + volFailuresTolerated
> + ". Value configured is either less than maxVolumeFailureLimit or
> greater than "
> + "to the number of configured volumes (" + volsConfigured + ").");{code}
>
> A {{DiskErrorException}} means an error has occurred when the process is
> interacting with the disk, e.g., in
> {{org.apache.hadoop.util.DiskChecker.checkDirInternal()}} we have the
> following code (lines 97-98):
> {code:java}
> throw new DiskErrorException("Cannot create directory: " +
> dir.toString());{code}
> However, the error message of the first exception indicates that
> {{dfs.datanode.failed.volumes.tolerated}} is configured incorrectly, which
> means there is nothing wrong with the disk (yet). This mismatch could be a
> problem. For example, the callers trying to handle other
> {{DiskErrorException}} may accidentally (and incorrectly) handle the
> configuration error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14401) Refine the implementation for HDFS cache on SCM
[ https://issues.apache.org/jira/browse/HDFS-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834352#comment-16834352 ] Feilong He commented on HDFS-14401: --- HDFS-14401.009.patch has been uploaded to address above issues reported by QA. > Refine the implementation for HDFS cache on SCM > --- > > Key: HDFS-14401 > URL: https://issues.apache.org/jira/browse/HDFS-14401 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, datanode >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Attachments: HDFS-14401.000.patch, HDFS-14401.001.patch, > HDFS-14401.002.patch, HDFS-14401.003.patch, HDFS-14401.004.patch, > HDFS-14401.005.patch, HDFS-14401.006.patch, HDFS-14401.007.patch, > HDFS-14401.008.patch, HDFS-14401.009.patch > > > In this Jira, we will refine the implementation for HDFS cache on SCM, such > as: 1) Handle full pmem volume in VolumeManager; 2) Refine pmem volume > selection impl; 3) Clean up MapppableBlockLoader interface; etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14401) Refine the implementation for HDFS cache on SCM
[ https://issues.apache.org/jira/browse/HDFS-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feilong He updated HDFS-14401: -- Attachment: HDFS-14401.009.patch > Refine the implementation for HDFS cache on SCM > --- > > Key: HDFS-14401 > URL: https://issues.apache.org/jira/browse/HDFS-14401 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, datanode >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Attachments: HDFS-14401.000.patch, HDFS-14401.001.patch, > HDFS-14401.002.patch, HDFS-14401.003.patch, HDFS-14401.004.patch, > HDFS-14401.005.patch, HDFS-14401.006.patch, HDFS-14401.007.patch, > HDFS-14401.008.patch, HDFS-14401.009.patch > > > In this Jira, we will refine the implementation for HDFS cache on SCM, such > as: 1) Handle full pmem volume in VolumeManager; 2) Refine pmem volume > selection impl; 3) Clean up MapppableBlockLoader interface; etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834347#comment-16834347
]
Hadoop QA commented on HDDS-1458:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 8m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
2s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
2s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
2s{color} | {color:blue} markdownlint was not available. {color} |
| {color:blue}0{color} | {color:blue} yamllint {color} | {color:blue} 0m
0s{color} | {color:blue} yamllint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 25 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 19m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} hadolint {color} | {color:red} 0m
2s{color} | {color:red} The patch generated 4 new + 0 unchanged - 0 fixed = 4
total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 17m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} pylint {color} | {color:green} 0m
29s{color} | {color:green} The patch generated 0 new + 768 unchanged - 7 fixed
= 768 total (was 775) {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
9s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
6s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 51s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}196m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HDDS-Build/2674/artifact/out/Dockerfile
|
| JIRA Issue | HDDS-1458 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12967990/HDDS-1458.004.patch |
| Optional Tests | dupname asflicense hadolint shellcheck
[jira] [Commented] (HDFS-13995) RBF: Security documentation
[
https://issues.apache.org/jira/browse/HDFS-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834324#comment-16834324
]
Brahma Reddy Battula commented on HDFS-13995:
-
Almost lgtm, how about adding like below for jute.maxbuffer..?
{code:java}
Since a large router/zookeeper cluster could potentially hold millions of
tokens and By default, ZooKeeper’s file size limit is 1MB. Attempting to write
or read files larger than this will cause errors.
ZooKeeper can be configured, via Java system property `jute.maxbuffer`, to
increase this limit. Note that this configuration, which is required both for
ZooKeeper server(s) and for all Routers that connect to the server(s), must be
the same everywhere it is specified.
{code}
> RBF: Security documentation
> ---
>
> Key: HDFS-13995
> URL: https://issues.apache.org/jira/browse/HDFS-13995
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-13995-HDFS-13891.001.patch,
> HDFS-13995-HDFS-13891.002.patch, HDFS-13995-HDFS-13891.003.patch
>
>
> Documentation for users under the section relating to security needs to be
> updated once security work is completely.
> [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html#Security]
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14134) Idempotent operations throwing RemoteException should not be retried by the client
[
https://issues.apache.org/jira/browse/HDFS-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834323#comment-16834323
]
Yuxuan Wang commented on HDFS-14134:
Hi [~lukmajercak]. No. I just figure that {{StandbyException}} always trigger
{{FAILOVER_AND_RETRY}}. And with hedging proxy, the action
{{FAILOVER_AND_RETRY}} will always be gotten and will cover all actions have
lower order than {{FAILOVER_AND_RETRY}}.
I mean that the order of {{enum RetryDecision}} may should be
{{FAILOVER_AND_RETRY < RETRY < FAIL}} in patch 007.
Or in short, {{FAILOVER_AND_RETRY}}'s order should be the lowest.
> Idempotent operations throwing RemoteException should not be retried by the
> client
> --
>
> Key: HDFS-14134
> URL: https://issues.apache.org/jira/browse/HDFS-14134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, hdfs-client, ipc
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
>Priority: Critical
> Attachments: HDFS-14134.001.patch, HDFS-14134.002.patch,
> HDFS-14134.003.patch, HDFS-14134.004.patch, HDFS-14134.005.patch,
> HDFS-14134.006.patch, HDFS-14134.007.patch,
> HDFS-14134_retrypolicy_change_proposal.pdf,
> HDFS-14134_retrypolicy_change_proposal_1.pdf
>
>
> Currently, some operations that throw IOException on the NameNode are
> evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail
> fast.
> For example, when calling getXAttr("user.some_attr", file") where the file
> does not have the attribute, NN throws an IOException with message "could not
> find attr". The current client retry policy determines the action for that to
> be FAILOVER_AND_RETRY. The client then fails over and retries until it
> reaches the maximum number of retries. Supposedly, the client should be able
> to tell that this exception is normal and fail fast.
> Moreover, even if the action was FAIL, the RetryInvocationHandler looks at
> all the retry actions from all requests, and FAILOVER_AND_RETRY takes
> precedence over FAIL action.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1467) Support configless Ozone service management
[ https://issues.apache.org/jira/browse/HDDS-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834319#comment-16834319 ] Eric Yang commented on HDDS-1467: - AWS and Azure both don't support multicast in their VPC environment. I don't think ZeroConf can be used on Cloud. It will only work in private data center. [This link|http://www.zeroconf.org/Rendezvous/txtrecords.html] provides good information on multicast dns TXT record format. A example code of using Multicast DNS to advertise ZooKeeper location can be found [here|https://github.com/macroadster/HMS/blob/master/beacon/src/main/java/org/apache/hms/beacon/Beacon.java]. > Support configless Ozone service management > --- > > Key: HDDS-1467 > URL: https://issues.apache.org/jira/browse/HDDS-1467 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: configless.pdf > > > The main goal is to simplify the configuration requirements for Ozone > components. > The idea is to merge the service discovery information (which is already > implemented in a simple form) and merge it with the current ozone-site.xml > and make it available for all the services. > Other services (such as OM, s3g) and client can download the current > configuration from SCM and register the host information. > Please see the attached design doc for more details. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14447) RBF: Router should support RefreshUserMappingsProtocol
[ https://issues.apache.org/jira/browse/HDFS-14447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834315#comment-16834315 ] Shen Yinjie commented on HDFS-14447: [~elgoiri][~crh] Thank you for comment,could you please review and help to push forward. > RBF: Router should support RefreshUserMappingsProtocol > -- > > Key: HDFS-14447 > URL: https://issues.apache.org/jira/browse/HDFS-14447 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Affects Versions: 3.1.0 >Reporter: Shen Yinjie >Assignee: Shen Yinjie >Priority: Major > Fix For: HDFS-13891 > > Attachments: HDFS-14447-HDFS-13891.01.patch, > HDFS-14447-HDFS-13891.02.patch, HDFS-14447-HDFS-13891.03.patch, > HDFS-14447-HDFS-13891.04.patch, HDFS-14447-HDFS-13891.05.patch, > HDFS-14447-HDFS-13891.06.patch, error.png > > > HDFS with RBF > We configure hadoop.proxyuser.xx.yy ,then execute hdfs dfsadmin > -Dfs.defaultFS=hdfs://router-fed -refreshSuperUserGroupsConfiguration, > it throws "Unknown protocol: ...RefreshUserMappingProtocol". > RouterAdminServer should support RefreshUserMappingsProtocol , or a proxyuser > client would be refused to impersonate.As shown in the screenshot -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13995) RBF: Security documentation
[ https://issues.apache.org/jira/browse/HDFS-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834311#comment-16834311 ] Akira Ajisaka commented on HDFS-13995: -- LGTM, +1. Thanks [~crh]! > RBF: Security documentation > --- > > Key: HDFS-13995 > URL: https://issues.apache.org/jira/browse/HDFS-13995 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-13995-HDFS-13891.001.patch, > HDFS-13995-HDFS-13891.002.patch, HDFS-13995-HDFS-13891.003.patch > > > Documentation for users under the section relating to security needs to be > updated once security work is completely. > [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html#Security] > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12510) RBF: Add security to UI
[
https://issues.apache.org/jira/browse/HDFS-12510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834307#comment-16834307
]
Brahma Reddy Battula commented on HDFS-12510:
-
{quote} Am fine either ways, but quite curious to know what the CORS issue is
wrt Router since router UI doesn't do any redirects.
{quote}
Even I feel same, I am not sure what ravi wants to be addressed. let's wait for
him.
{quote}Created a new jira HDFS-14475 to track the work earlier done here.
{quote}
thanks.
> RBF: Add security to UI
> ---
>
> Key: HDFS-12510
> URL: https://issues.apache.org/jira/browse/HDFS-12510
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Labels: RBF
>
> HDFS-12273 implemented the UI for Router Based Federation without security.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14475) RBF: Expose router security enabled status on the UI
[
https://issues.apache.org/jira/browse/HDFS-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834305#comment-16834305
]
Brahma Reddy Battula commented on HDFS-14475:
-
[~crh] thanks for reporting this.
IMO, Following also can be removed , as this might not used. FYI, this is
introduced by HDFS-13466.
Path :
(\hadoop-hdfs-project\hadoop-hdfs-rbf\src\main\webapps\router\federationhealth.js)
{code:java}
{"name": "routerstat", "url":
"/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus"},{code}
> RBF: Expose router security enabled status on the UI
>
>
> Key: HDFS-14475
> URL: https://issues.apache.org/jira/browse/HDFS-14475
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14475-HDFS-13891.001.patch
>
>
> This is a branched off Jira to expose metric so that router's security status
> can be displayed on the UI. We are still unclear if more work needs to be
> done for dealing with CORS etc.
> https://issues.apache.org/jira/browse/HDFS-12510 will continue to track that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1491) Ozone KeyInputStream seek() should not read the chunk file
[ https://issues.apache.org/jira/browse/HDDS-1491?focusedWorklogId=238229=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238229 ] ASF GitHub Bot logged work on HDDS-1491: Author: ASF GitHub Bot Created on: 07/May/19 00:35 Start Date: 07/May/19 00:35 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #795: HDDS-1491. Ozone KeyInputStream seek() should not read the chunk file. URL: https://github.com/apache/hadoop/pull/795#issuecomment-489861096 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 50 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 399 | trunk passed | | +1 | compile | 199 | trunk passed | | +1 | checkstyle | 53 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 835 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 134 | trunk passed | | 0 | spotbugs | 234 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 412 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 412 | the patch passed | | +1 | compile | 290 | the patch passed | | +1 | javac | 290 | the patch passed | | +1 | checkstyle | 146 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 1162 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 325 | the patch passed | | +1 | findbugs | 668 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 162 | hadoop-hdds in the patch failed. | | -1 | unit | 1302 | hadoop-ozone in the patch failed. | | +1 | asflicense | 51 | The patch does not generate ASF License warnings. | | | | 6795 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.container.TestContainerReplication | | | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | | hadoop.ozone.client.rpc.TestReadRetries | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-795/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/795 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux db6e5c7123f2 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 597fa47 | | Default Java | 1.8.0_191 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-795/2/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-795/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-795/2/testReport/ | | Max. process+thread count | 3555 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/client U: hadoop-hdds/client | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-795/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 238229) Time Spent: 0.5h (was: 20m) > Ozone KeyInputStream seek() should not read the chunk file > -- > > Key: HDDS-1491 > URL: https://issues.apache.org/jira/browse/HDDS-1491 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels:
[jira] [Work logged] (HDDS-1489) Unnecessary log messages on console with Ozone shell
[ https://issues.apache.org/jira/browse/HDDS-1489?focusedWorklogId=238224=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238224 ] ASF GitHub Bot logged work on HDDS-1489: Author: ASF GitHub Bot Created on: 07/May/19 00:10 Start Date: 07/May/19 00:10 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #797: HDDS-1489. Unnecessary log messages on console with Ozone shell. URL: https://github.com/apache/hadoop/pull/797#issuecomment-489831647 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 33 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | 0 | shelldocs | 0 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 48 | Maven dependency ordering for branch | | +1 | mvninstall | 433 | trunk passed | | +1 | compile | 211 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 740 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 131 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 16 | Maven dependency ordering for patch | | +1 | mvninstall | 426 | the patch passed | | +1 | compile | 208 | the patch passed | | +1 | javac | 208 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 0 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 640 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 124 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 141 | hadoop-hdds in the patch failed. | | -1 | unit | 952 | hadoop-ozone in the patch failed. | | +1 | asflicense | 47 | The patch does not generate ASF License warnings. | | | | 4322 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider | | | hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-797/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/797 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient shellcheck shelldocs | | uname | Linux 949ab61dfba0 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 93f2283 | | Default Java | 1.8.0_191 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-797/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-797/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-797/1/testReport/ | | Max. process+thread count | 4370 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/dist hadoop-ozone/ozonefs U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-797/1/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 238224) Time Spent: 1h (was: 50m) > Unnecessary log messages on console with Ozone shell > - > > Key: HDDS-1489 > URL: https://issues.apache.org/jira/browse/HDDS-1489 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone CLI >Reporter: Arpit Agarwal >Assignee: Siddharth Wagle >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > The following log messages are
[jira] [Comment Edited] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[ https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834199#comment-16834199 ] Chen Liang edited comment on HDFS-12979 at 5/7/19 12:10 AM: Post v003 patch which uses the alternative approach of having more than one primary checkpointers. Namely, different ANN/Observer can have different SbN as their primary. Upside is that no need to add the new http code, and the image uploading workload gets distributed across SbNs; Downside is that it seems ANN/Observer could have different images on their disk because of different primary check pointers, so when system restart, there is a possibility that they may pick up different images then ANN/Observer could potentially run into different txids. Which may or may not be a problem. Still need to verify this, if yes, we need to be careful about this. Any thoughts [~shv]? UPDATE: Actually, I think the v002 patch approach also has the potential problem of observer could have a different version of image from ANN. So this is a potential issue either way. was (Author: vagarychen): Post v003 patch which uses the alternative approach of having more than one primary checkpointers. Namely, different ANN/Observer can have different SbN as their primary. Upside is that no need to add the new http code, and the image uploading workload gets distributed across SbNs; Downside is that it seems ANN/Observer could have different images on their disk because of different primary check pointers, so when system restart, there is a possibility that they may pick up different images then ANN/Observer could potentially run into different txids. Which may or may not be a problem. Still need to verify this, if yes, we need to be careful about this. Any thoughts [~shv]? > StandbyNode should upload FsImage to ObserverNode after checkpointing. > -- > > Key: HDFS-12979 > URL: https://issues.apache.org/jira/browse/HDFS-12979 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Konstantin Shvachko >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-12979.001.patch, HDFS-12979.002.patch, > HDFS-12979.003.patch > > > ObserverNode does not create checkpoints. So it's fsimage file can get very > old making bootstrap of ObserverNode too long. A StandbyNode should copy > latest fsimage to ObserverNode(s) along with ANN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834291#comment-16834291
]
Hadoop QA commented on HDDS-1458:
-
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://builds.apache.org/job/PreCommit-HDDS-Build/2674/console in case of
problems.
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HDDS-1458.001.patch, HDDS-1458.002.patch,
> HDDS-1458.003.patch, HDDS-1458.004.patch
>
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1474) "ozone.scm.datanode.id" config should take path for a dir and not a file
[ https://issues.apache.org/jira/browse/HDDS-1474?focusedWorklogId=238222=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238222 ] ASF GitHub Bot logged work on HDDS-1474: Author: ASF GitHub Bot Created on: 07/May/19 00:04 Start Date: 07/May/19 00:04 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #792: HDDS-1474. ozone.scm.datanode.id config should take path for a dir URL: https://github.com/apache/hadoop/pull/792#issuecomment-489830654 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1045 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | 0 | shelldocs | 0 | Shelldocs was not available. | | 0 | yamllint | 0 | yamllint was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for branch | | +1 | mvninstall | 488 | trunk passed | | +1 | compile | 233 | trunk passed | | +1 | checkstyle | 64 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 905 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 149 | trunk passed | | 0 | spotbugs | 306 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 526 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 32 | Maven dependency ordering for patch | | +1 | mvninstall | 487 | the patch passed | | +1 | compile | 216 | the patch passed | | +1 | javac | 216 | the patch passed | | +1 | checkstyle | 63 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 0 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 729 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 126 | the patch passed | | +1 | findbugs | 450 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 153 | hadoop-hdds in the patch failed. | | -1 | unit | 1243 | hadoop-ozone in the patch failed. | | +1 | asflicense | 37 | The patch does not generate ASF License warnings. | | | | 7204 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.TestContainerReplication | | | hadoop.ozone.client.rpc.TestCommitWatcher | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-792/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/792 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle shellcheck shelldocs yamllint | | uname | Linux 964887b87752 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 597fa47 | | Default Java | 1.8.0_191 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-792/5/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-792/5/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-792/5/testReport/ | | Max. process+thread count | 4195 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/container-service hadoop-hdds/docs hadoop-ozone/dist U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-792/5/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 238222) Time Spent: 1h 50m (was: 1h 40m) > "ozone.scm.datanode.id" config should take path for a dir and not a file > > > Key: HDDS-1474 > URL: https://issues.apache.org/jira/browse/HDDS-1474 > Project: Hadoop
[jira] [Updated] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDDS-1458:
Attachment: HDDS-1458.004.patch
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HDDS-1458.001.patch, HDDS-1458.002.patch,
> HDDS-1458.003.patch, HDDS-1458.004.patch
>
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDDS-1458:
Attachment: (was: HDDS-1458.004.patch)
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HDDS-1458.001.patch, HDDS-1458.002.patch,
> HDDS-1458.003.patch
>
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[
https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834267#comment-16834267
]
Hadoop QA commented on HDFS-12979:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 8 new + 76 unchanged - 3 fixed = 84 total (was 79) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
10s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0
unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 4s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| |
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.checkpointReceivers
is or uses a map or set of URLs, which can be a performance hog In
StandbyCheckpointer.java:map or set of URLs, which can be a performance hog In
StandbyCheckpointer.java |
| |
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.doCheckpoint(boolean)
is or uses a map or set of URLs, which can be a performance hog At
StandbyCheckpointer.java:map or set of URLs, which can be a performance hog At
StandbyCheckpointer.java:[line 247] |
| |
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.doCheckpoint(boolean)
is or uses a map or set of URLs, which can be a performance hog At
StandbyCheckpointer.java:map or set of URLs, which can be a performance hog At
StandbyCheckpointer.java:[line 236] |
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-12979 |
| JIRA Patch URL |
[jira] [Work logged] (HDDS-1489) Unnecessary log messages on console with Ozone shell
[
https://issues.apache.org/jira/browse/HDDS-1489?focusedWorklogId=238195=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238195
]
ASF GitHub Bot logged work on HDDS-1489:
Author: ASF GitHub Bot
Created on: 06/May/19 23:09
Start Date: 06/May/19 23:09
Worklog Time Spent: 10m
Work Description: shwetayakkali commented on pull request #797:
HDDS-1489. Unnecessary log messages on console with Ozone shell.
URL: https://github.com/apache/hadoop/pull/797#discussion_r281399158
##
File path: hadoop-ozone/dist/src/main/compose/ozoneperf/docker-config
##
@@ -36,3 +36,4 @@
LOG4J.PROPERTIES_log4j.appender.stdout.layout.ConversionPattern=%d{-MM-dd HH
LOG4J.PROPERTIES_log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
LOG4J.PROPERTIES_log4j.logger.org.apache.ratis.conf.ConfUtils=WARN
LOG4J.PROPERTIES_log4j.logger.org.apache.hadoop.security.ShellBasedUnixGroupsMapping=ERROR
+LOG4J.PROPERTIES_log4j.logger.org.apache.ratis.grpc.client.GrpcClientProtocolClient=WARN
Review comment:
checkstyle warning here to add a new line at the end of file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 238195)
Time Spent: 50m (was: 40m)
> Unnecessary log messages on console with Ozone shell
> -
>
> Key: HDDS-1489
> URL: https://issues.apache.org/jira/browse/HDDS-1489
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Reporter: Arpit Agarwal
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The following log messages are printed on the console when running putkey
> {code}
> $ ozone sh key put /vol1/bucket1/key1 myfile
> 2019-05-03 23:25:15 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=0, SUCCESS, logIndex=1, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c2]
> 2019-05-03 23:25:16 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=1, SUCCESS, logIndex=3, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c4]
> {code}
> These are unnecessary noise and should be suppressed by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDDS-1458:
Attachment: HDDS-1458.004.patch
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HDDS-1458.001.patch, HDDS-1458.002.patch,
> HDDS-1458.003.patch, HDDS-1458.004.patch
>
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1489) Unnecessary log messages on console with Ozone shell
[
https://issues.apache.org/jira/browse/HDDS-1489?focusedWorklogId=238193=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238193
]
ASF GitHub Bot logged work on HDDS-1489:
Author: ASF GitHub Bot
Created on: 06/May/19 23:07
Start Date: 06/May/19 23:07
Worklog Time Spent: 10m
Work Description: swagle commented on issue #797: HDDS-1489. Unnecessary
log messages on console with Ozone shell.
URL: https://github.com/apache/hadoop/pull/797#issuecomment-489819230
@arp7 To change the log level of the message would need a Ratis Jira since
it is,
org.apache.ratis.grpc.client.GrpcClientProtocolClient.AsyncStreamObservers,
logging this message.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 238193)
Time Spent: 0.5h (was: 20m)
> Unnecessary log messages on console with Ozone shell
> -
>
> Key: HDDS-1489
> URL: https://issues.apache.org/jira/browse/HDDS-1489
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Reporter: Arpit Agarwal
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The following log messages are printed on the console when running putkey
> {code}
> $ ozone sh key put /vol1/bucket1/key1 myfile
> 2019-05-03 23:25:15 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=0, SUCCESS, logIndex=1, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c2]
> 2019-05-03 23:25:16 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=1, SUCCESS, logIndex=3, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c4]
> {code}
> These are unnecessary noise and should be suppressed by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1474) "ozone.scm.datanode.id" config should take path for a dir and not a file
[ https://issues.apache.org/jira/browse/HDDS-1474?focusedWorklogId=238197=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238197 ] ASF GitHub Bot logged work on HDDS-1474: Author: ASF GitHub Bot Created on: 06/May/19 23:12 Start Date: 06/May/19 23:12 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #792: HDDS-1474. ozone.scm.datanode.id config should take path for a dir URL: https://github.com/apache/hadoop/pull/792#issuecomment-489820176 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 509 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | 0 | shelldocs | 0 | Shelldocs was not available. | | 0 | yamllint | 0 | yamllint was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 81 | Maven dependency ordering for branch | | +1 | mvninstall | 422 | trunk passed | | +1 | compile | 210 | trunk passed | | +1 | checkstyle | 59 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 771 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 134 | trunk passed | | 0 | spotbugs | 238 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 426 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 33 | Maven dependency ordering for patch | | +1 | mvninstall | 406 | the patch passed | | +1 | compile | 219 | the patch passed | | +1 | javac | 219 | the patch passed | | -0 | checkstyle | 31 | hadoop-hdds: The patch generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 0 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 650 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 132 | the patch passed | | +1 | findbugs | 437 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 130 | hadoop-hdds in the patch failed. | | -1 | unit | 928 | hadoop-ozone in the patch failed. | | +1 | asflicense | 43 | The patch does not generate ASF License warnings. | | | | 5879 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-792/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/792 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle shellcheck shelldocs yamllint | | uname | Linux 618db41f1fc3 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 597fa47 | | Default Java | 1.8.0_191 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-792/4/artifact/out/diff-checkstyle-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-792/4/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-792/4/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-792/4/testReport/ | | Max. process+thread count | 4471 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/container-service hadoop-hdds/docs hadoop-ozone/dist U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-792/4/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 238197) Time Spent: 1h 40m (was: 1.5h) > "ozone.scm.datanode.id" config should take path for a dir and not a file >
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834263#comment-16834263
]
Eric Yang commented on HDDS-1458:
-
[~elek] Patch 004 will allow both type of docker container to work. The
requisite for development docker image is to make sure that dist project is
built with target/ozone-${project.version} directory available for mounting.
Command line usage for development test (with hadoop-runner image):
{code}
mvn clean verify -Pit
{code}
Command line usage for binary distribution test (with
apache/ozone-${project.version} image):
{code}
mvn clean verify -Pit,dist
{code}
Does this arrangement work for you?
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HDDS-1458.001.patch, HDDS-1458.002.patch,
> HDDS-1458.003.patch, HDDS-1458.004.patch
>
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1489) Unnecessary log messages on console with Ozone shell
[
https://issues.apache.org/jira/browse/HDDS-1489?focusedWorklogId=238194=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238194
]
ASF GitHub Bot logged work on HDDS-1489:
Author: ASF GitHub Bot
Created on: 06/May/19 23:08
Start Date: 06/May/19 23:08
Worklog Time Spent: 10m
Work Description: shwetayakkali commented on pull request #797:
HDDS-1489. Unnecessary log messages on console with Ozone shell.
URL: https://github.com/apache/hadoop/pull/797#discussion_r281399103
##
File path: hadoop-ozone/dist/src/main/compose/ozonefs/docker-config
##
@@ -35,3 +35,4 @@
LOG4J.PROPERTIES_log4j.appender.stdout=org.apache.log4j.ConsoleAppender
LOG4J.PROPERTIES_log4j.appender.stdout.layout.ConversionPattern=%d{-MM-dd
HH:mm:ss} %-5p %c{1}:%L - %m%n
LOG4J.PROPERTIES_log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
LOG4J.PROPERTIES_log4j.logger.org.apache.ratis.conf.ConfUtils=WARN
+LOG4J.PROPERTIES_log4j.logger.org.apache.ratis.grpc.client.GrpcClientProtocolClient=WARN
Review comment:
checkstyle warning to add a new line at the end of file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 238194)
Time Spent: 40m (was: 0.5h)
> Unnecessary log messages on console with Ozone shell
> -
>
> Key: HDDS-1489
> URL: https://issues.apache.org/jira/browse/HDDS-1489
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Reporter: Arpit Agarwal
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The following log messages are printed on the console when running putkey
> {code}
> $ ozone sh key put /vol1/bucket1/key1 myfile
> 2019-05-03 23:25:15 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=0, SUCCESS, logIndex=1, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c2]
> 2019-05-03 23:25:16 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=1, SUCCESS, logIndex=3, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c4]
> {code}
> These are unnecessary noise and should be suppressed by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1489) Unnecessary log messages on console with Ozone shell
[
https://issues.apache.org/jira/browse/HDDS-1489?focusedWorklogId=238188=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238188
]
ASF GitHub Bot logged work on HDDS-1489:
Author: ASF GitHub Bot
Created on: 06/May/19 22:56
Start Date: 06/May/19 22:56
Worklog Time Spent: 10m
Work Description: swagle commented on pull request #797: HDDS-1489.
Unnecessary log messages on console with Ozone shell.
URL: https://github.com/apache/hadoop/pull/797
cc: @arp7
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 238188)
Time Spent: 10m
Remaining Estimate: 0h
> Unnecessary log messages on console with Ozone shell
> -
>
> Key: HDDS-1489
> URL: https://issues.apache.org/jira/browse/HDDS-1489
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Reporter: Arpit Agarwal
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following log messages are printed on the console when running putkey
> {code}
> $ ozone sh key put /vol1/bucket1/key1 myfile
> 2019-05-03 23:25:15 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=0, SUCCESS, logIndex=1, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c2]
> 2019-05-03 23:25:16 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=1, SUCCESS, logIndex=3, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c4]
> {code}
> These are unnecessary noise and should be suppressed by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1489) Unnecessary log messages on console with Ozone shell
[
https://issues.apache.org/jira/browse/HDDS-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-1489:
-
Labels: pull-request-available (was: )
> Unnecessary log messages on console with Ozone shell
> -
>
> Key: HDDS-1489
> URL: https://issues.apache.org/jira/browse/HDDS-1489
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Reporter: Arpit Agarwal
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: pull-request-available
>
> The following log messages are printed on the console when running putkey
> {code}
> $ ozone sh key put /vol1/bucket1/key1 myfile
> 2019-05-03 23:25:15 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=0, SUCCESS, logIndex=1, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c2]
> 2019-05-03 23:25:16 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=1, SUCCESS, logIndex=3, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c4]
> {code}
> These are unnecessary noise and should be suppressed by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1489) Unnecessary log messages on console with Ozone shell
[
https://issues.apache.org/jira/browse/HDDS-1489?focusedWorklogId=238189=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238189
]
ASF GitHub Bot logged work on HDDS-1489:
Author: ASF GitHub Bot
Created on: 06/May/19 22:57
Start Date: 06/May/19 22:57
Worklog Time Spent: 10m
Work Description: arp7 commented on issue #797: HDDS-1489. Unnecessary
log messages on console with Ozone shell.
URL: https://github.com/apache/hadoop/pull/797#issuecomment-489817144
Instead of setting log threshold to WARN, would it be better to move the
offending log messsages to DEBUG level?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 238189)
Time Spent: 20m (was: 10m)
> Unnecessary log messages on console with Ozone shell
> -
>
> Key: HDDS-1489
> URL: https://issues.apache.org/jira/browse/HDDS-1489
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Reporter: Arpit Agarwal
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The following log messages are printed on the console when running putkey
> {code}
> $ ozone sh key put /vol1/bucket1/key1 myfile
> 2019-05-03 23:25:15 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=0, SUCCESS, logIndex=1, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c2]
> 2019-05-03 23:25:16 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=1, SUCCESS, logIndex=3, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c4]
> {code}
> These are unnecessary noise and should be suppressed by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1489) Unnecessary log messages on console with Ozone shell
[
https://issues.apache.org/jira/browse/HDDS-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Wagle updated HDDS-1489:
--
Status: Patch Available (was: Open)
> Unnecessary log messages on console with Ozone shell
> -
>
> Key: HDDS-1489
> URL: https://issues.apache.org/jira/browse/HDDS-1489
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Reporter: Arpit Agarwal
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following log messages are printed on the console when running putkey
> {code}
> $ ozone sh key put /vol1/bucket1/key1 myfile
> 2019-05-03 23:25:15 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=0, SUCCESS, logIndex=1, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c2]
> 2019-05-03 23:25:16 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=1, SUCCESS, logIndex=3, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c4]
> {code}
> These are unnecessary noise and should be suppressed by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14245) Class cast error in GetGroups with ObserverReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834245#comment-16834245
]
Konstantin Shvachko edited comment on HDFS-14245 at 5/6/19 10:53 PM:
-
??getHAServiceState() assumes STANDBY state no matter what error
I agree re-throwing wont be good here. But why not return NULL as service
state. It is used to set as cached value. Having cached value as NULL with a
meaning the state is not cached would be logical.
Otherwise the whole unwrapping of {{RemoteException}} is redundant since it is
an {{IOException}} as well.
The rest looks good.
was (Author: shv):
??getHAServiceState() assumes STANDBY state no matter what error
I agree re-throwing wont be good here. But why not return NULL as service
state. It used to set as cached value. Having cached value as NULL with a
meaning the state is not cached would be logical.
Otherwise the whole unwrapping of {{RemoteException}} is redundant since it is
an {{IOException}} as well.
The rest looks good.
> Class cast error in GetGroups with ObserverReadProxyProvider
>
>
> Key: HDFS-14245
> URL: https://issues.apache.org/jira/browse/HDFS-14245
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: HDFS-12943
>Reporter: Shen Yinjie
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14245.000.patch, HDFS-14245.001.patch,
> HDFS-14245.002.patch, HDFS-14245.003.patch, HDFS-14245.004.patch,
> HDFS-14245.patch
>
>
> Run "hdfs groups" with ObserverReadProxyProvider, Exception throws as :
> {code:java}
> Exception in thread "main" java.io.IOException: Couldn't create proxy
> provider class
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:261)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:119)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:95)
> at org.apache.hadoop.hdfs.tools.GetGroups.getUgmProtocol(GetGroups.java:87)
> at org.apache.hadoop.tools.GetGroupsBase.run(GetGroupsBase.java:71)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.GetGroups.main(GetGroups.java:96)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:245)
> ... 7 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.namenode.ha.NameNodeHAProxyFactory cannot be
> cast to org.apache.hadoop.hdfs.server.namenode.ha.ClientHAProxyFactory
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider.(ObserverReadProxyProvider.java:123)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider.(ObserverReadProxyProvider.java:112)
> ... 12 more
> {code}
> similar with HDFS-14116, we did a simple fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14245) Class cast error in GetGroups with ObserverReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834245#comment-16834245
]
Konstantin Shvachko edited comment on HDFS-14245 at 5/6/19 10:43 PM:
-
??getHAServiceState() assumes STANDBY state no matter what error
I agree re-throwing wont be good here. But why not return NULL as service
state. It used to set as cached value. Having cached value as NULL with a
meaning the state is not cached would be logical.
Otherwise the whole unwrapping of {{RemoteException}} is redundant since it is
an {{IOException}} as well.
The rest looks good.
was (Author: shv):
??getHAServiceState() assumes STANDBY state no matter what error
I agree re-throwing wont be good here. But why not return NULL as service
state. It used to set as cached value. Having cached value as NULL with a
meaning the state is not cached would be logical.
Otherwise the whole unwrapping of {{RemoteException}} is redundant since it is
an {{IOException}} as well.
> Class cast error in GetGroups with ObserverReadProxyProvider
>
>
> Key: HDFS-14245
> URL: https://issues.apache.org/jira/browse/HDFS-14245
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: HDFS-12943
>Reporter: Shen Yinjie
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14245.000.patch, HDFS-14245.001.patch,
> HDFS-14245.002.patch, HDFS-14245.003.patch, HDFS-14245.004.patch,
> HDFS-14245.patch
>
>
> Run "hdfs groups" with ObserverReadProxyProvider, Exception throws as :
> {code:java}
> Exception in thread "main" java.io.IOException: Couldn't create proxy
> provider class
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:261)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:119)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:95)
> at org.apache.hadoop.hdfs.tools.GetGroups.getUgmProtocol(GetGroups.java:87)
> at org.apache.hadoop.tools.GetGroupsBase.run(GetGroupsBase.java:71)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.GetGroups.main(GetGroups.java:96)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:245)
> ... 7 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.namenode.ha.NameNodeHAProxyFactory cannot be
> cast to org.apache.hadoop.hdfs.server.namenode.ha.ClientHAProxyFactory
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider.(ObserverReadProxyProvider.java:123)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider.(ObserverReadProxyProvider.java:112)
> ... 12 more
> {code}
> similar with HDFS-14116, we did a simple fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14245) Class cast error in GetGroups with ObserverReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834245#comment-16834245
]
Konstantin Shvachko commented on HDFS-14245:
??getHAServiceState() assumes STANDBY state no matter what error
I agree re-throwing wont be good here. But why not return NULL as service
state. It used to set as cached value. Having cached value as NULL with a
meaning the state is not cached would be logical.
Otherwise the whole unwrapping of {{RemoteException}} is redundant since it is
an {{IOException}} as well.
> Class cast error in GetGroups with ObserverReadProxyProvider
>
>
> Key: HDFS-14245
> URL: https://issues.apache.org/jira/browse/HDFS-14245
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: HDFS-12943
>Reporter: Shen Yinjie
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14245.000.patch, HDFS-14245.001.patch,
> HDFS-14245.002.patch, HDFS-14245.003.patch, HDFS-14245.004.patch,
> HDFS-14245.patch
>
>
> Run "hdfs groups" with ObserverReadProxyProvider, Exception throws as :
> {code:java}
> Exception in thread "main" java.io.IOException: Couldn't create proxy
> provider class
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:261)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:119)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:95)
> at org.apache.hadoop.hdfs.tools.GetGroups.getUgmProtocol(GetGroups.java:87)
> at org.apache.hadoop.tools.GetGroupsBase.run(GetGroupsBase.java:71)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.GetGroups.main(GetGroups.java:96)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:245)
> ... 7 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.namenode.ha.NameNodeHAProxyFactory cannot be
> cast to org.apache.hadoop.hdfs.server.namenode.ha.ClientHAProxyFactory
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider.(ObserverReadProxyProvider.java:123)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider.(ObserverReadProxyProvider.java:112)
> ... 12 more
> {code}
> similar with HDFS-14116, we did a simple fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14426) RBF: Add delegation token total count as one of the federation metrics
[
https://issues.apache.org/jira/browse/HDFS-14426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834237#comment-16834237
]
Hadoop QA commented on HDFS-14426:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
11s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 23m
28s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14426 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12967972/HDFS-14426-HDFS-13891.005.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5549c14082ff 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / 206b082 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26757/testReport/ |
| Max. process+thread count | 998 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26757/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Add delegation token total count as one of the federation metrics
>
[jira] [Commented] (HDDS-1495) Create hadoop/ozone docker images with inline build process
[
https://issues.apache.org/jira/browse/HDDS-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834223#comment-16834223
]
Hadoop QA commented on HDDS-1495:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
1s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 5m
40s{color} | {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} hadolint {color} | {color:red} 0m
1s{color} | {color:red} The patch generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
7s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 56s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
20s{color} | {color:red} hadoop-ozone in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 53s{color}
| {color:red} hadoop-hdds in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 48s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient |
| | hadoop.ozone.om.TestScmSafeMode |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HDDS-Build/2673/artifact/out/Dockerfile
|
| JIRA Issue | HDDS-1495 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[ https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834199#comment-16834199 ] Chen Liang commented on HDFS-12979: --- Post v003 patch which uses the alternative approach of having more than one primary checkpointers. Namely, different ANN/Observer can have different SbN as their primary. Upside is that no need to add the new http code, and the image uploading workload gets distributed across SbNs; Downside is that it seems ANN/Observer could have different images on their disk because of different primary check pointers, so when system restart, there is a possibility that they may pick up different images then ANN/Observer could potentially run into different txids. Which may or may not be a problem. Still need to verify this, if yes, we need to be careful about this. Any thoughts [~shv]? > StandbyNode should upload FsImage to ObserverNode after checkpointing. > -- > > Key: HDFS-12979 > URL: https://issues.apache.org/jira/browse/HDFS-12979 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Konstantin Shvachko >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-12979.001.patch, HDFS-12979.002.patch, > HDFS-12979.003.patch > > > ObserverNode does not create checkpoints. So it's fsimage file can get very > old making bootstrap of ObserverNode too long. A StandbyNode should copy > latest fsimage to ObserverNode(s) along with ANN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[ https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12979: -- Attachment: HDFS-12979.003.patch > StandbyNode should upload FsImage to ObserverNode after checkpointing. > -- > > Key: HDFS-12979 > URL: https://issues.apache.org/jira/browse/HDFS-12979 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Konstantin Shvachko >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-12979.001.patch, HDFS-12979.002.patch, > HDFS-12979.003.patch > > > ObserverNode does not create checkpoints. So it's fsimage file can get very > old making bootstrap of ObserverNode too long. A StandbyNode should copy > latest fsimage to ObserverNode(s) along with ANN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1424) Support multi-container robot test execution
[
https://issues.apache.org/jira/browse/HDDS-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834196#comment-16834196
]
Arpit Agarwal commented on HDDS-1424:
-
I ran a few of the smoke tests locally and they passed on the first attempt. I
think some of the previous flakiness is gone.
Big +1 for these changes. Had a couple of minor comments on the PR, not
blockers to commit.
> Support multi-container robot test execution
>
>
> Key: HDDS-1424
> URL: https://issues.apache.org/jira/browse/HDDS-1424
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> The ./smoketest folder in the distribution package contains robotframework
> based test scripts to test the main behaviour of Ozone.
> The tests have two layers:
> 1. robot test definitions to execute commands and assert the results (on a
> given host machine)
> 2. ./smoketest/test.sh which starts/stops the docker-compose based
> environments AND execute the selected robot tests inside the right hosts
> The second one (test.sh) has some serious limitations:
> 1. all the tests are executed inside the same container (om):
> https://github.com/apache/hadoop/blob/5f951ea2e39ae4dfe554942baeec05849cd7d3c2/hadoop-ozone/dist/src/main/smoketest/test.sh#L89
> Some of the tests (ozonesecure-mr, ozonefs) may require the flexibility to
> execute different robot tests in different containers.
> 2. The definition of the global test set is complex and hard to understood.
> The current code is:
> {code}
>TESTS=("basic")
>execute_tests ozone "${TESTS[@]}"
>TESTS=("auditparser")
>execute_tests ozone "${TESTS[@]}"
>TESTS=("ozonefs")
>execute_tests ozonefs "${TESTS[@]}"
>TESTS=("basic")
>execute_tests ozone-hdfs "${TESTS[@]}"
>TESTS=("s3")
>execute_tests ozones3 "${TESTS[@]}"
>TESTS=("security")
>execute_tests ozonesecure .
> {code}
> For example for ozonesecure the TESTS is not used. And the usage of bash
> lists require additional complexity in the execute_tests function.
> I propose here a very lightweight refactor. Instead of including both the
> test definitions AND the helper methods in test.sh I would separate them.
> Let's put a test.sh to each of the compose directories. The separated test.sh
> can include common methods from a main shell script. For example:
> {code}
> source "$COMPOSE_DIR/../testlib.sh"
> start_docker_env
> execute_robot_test scm basic/basic.robot
> execute_robot_test scm s3
> stop_docker_env
> generate_report
> {code}
> This is a more clean and more flexible definition. It's easy to execute just
> this test (as it's saved to the compose/ozones3 directory. And it's more
> flexible.
> Other example, where multiple containers are used to execute tests:
> {code}
> source "$COMPOSE_DIR/../testlib.sh"
> start_docker_env
> execute_robot_test scm ozonefs/ozonefs.robot
> export OZONE_HOME=/opt/ozone
> execute_robot_test hadoop32 ozonefs/hadoopo3fs.robot
> execute_robot_test hadoop31 ozonefs/hadoopo3fs.robot
> stop_docker_env
> generate_report
> {code}
> With this separation the definition of the helper methods (eg.
> execute_robot_test or stop_docker_env) would also be simplified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[
https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834195#comment-16834195
]
Hadoop QA commented on HDFS-12979:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 91 unchanged - 1 fixed = 92 total (was 92) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 51s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-12979 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12967966/HDFS-12979.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c70a9cd51b02 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fb7c1ca |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26756/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26756/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26756/testReport/ |
| Max. process+thread count | 4634 (vs. ulimit of 1) |
| modules | C:
[jira] [Assigned] (HDFS-14461) RBF: Fix intermittently failing kerberos related unit test
[
https://issues.apache.org/jira/browse/HDFS-14461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fengnan Li reassigned HDFS-14461:
-
Assignee: Fengnan Li
> RBF: Fix intermittently failing kerberos related unit test
> --
>
> Key: HDFS-14461
> URL: https://issues.apache.org/jira/browse/HDFS-14461
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: Fengnan Li
>Priority: Major
>
> TestRouterHttpDelegationToken#testGetDelegationToken fails intermittently. It
> may be due to some race condition before using the keytab that's created for
> testing.
>
> {code:java}
> Failed
> org.apache.hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken.testGetDelegationToken
> Failing for the past 1 build (Since
> [!https://builds.apache.org/static/1e9ab9cc/images/16x16/red.png!
> #26721|https://builds.apache.org/job/PreCommit-HDFS-Build/26721/] )
> [Took 89
> ms.|https://builds.apache.org/job/PreCommit-HDFS-Build/26721/testReport/org.apache.hadoop.hdfs.server.federation.security/TestRouterHttpDelegationToken/testGetDelegationToken/history]
>
> Error Message
> org.apache.hadoop.security.KerberosAuthException: failure to login: for
> principal: router/localh...@example.com from keytab
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-rbf/target/test/data/SecurityConfUtil/test.keytab
> javax.security.auth.login.LoginException: Integrity check on decrypted field
> failed (31) - PREAUTH_FAILED
> h3. Stacktrace
> org.apache.hadoop.service.ServiceStateException:
> org.apache.hadoop.security.KerberosAuthException: failure to login: for
> principal: router/localh...@example.com from keytab
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-rbf/target/test/data/SecurityConfUtil/test.keytab
> javax.security.auth.login.LoginException: Integrity check on decrypted field
> failed (31) - PREAUTH_FAILED at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:173)
> at
> org.apache.hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken.setup(TestRouterHttpDelegationToken.java:99)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by: org.apache.hadoop.security.KerberosAuthException: failure to
> login: for principal: router/localh...@example.com from keytab
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-rbf/target/test/data/SecurityConfUtil/test.keytab
> javax.security.auth.login.LoginException: Integrity check on decrypted field
> failed (31) - PREAUTH_FAILED at
> org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2008)
> at
[jira] [Commented] (HDFS-14426) RBF: Add delegation token total count as one of the federation metrics
[ https://issues.apache.org/jira/browse/HDFS-14426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834183#comment-16834183 ] Fengnan Li commented on HDFS-14426: --- [^HDFS-14426-HDFS-13891.003.patch] is exactly the same as [^HDFS-14426-HDFS-13891.003.patch] Just want to trigger the build again. > RBF: Add delegation token total count as one of the federation metrics > -- > > Key: HDFS-14426 > URL: https://issues.apache.org/jira/browse/HDFS-14426 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Attachments: HDFS-14426-HDFS-13891.001.patch, > HDFS-14426-HDFS-13891.002.patch, HDFS-14426-HDFS-13891.003.patch, > HDFS-14426-HDFS-13891.004.patch, HDFS-14426-HDFS-13891.005.patch, > HDFS-14426.001.patch > > > Currently router doesn't report the total number of current valid delegation > tokens it has, but this piece of information is useful for monitoring and > understanding the real time situation of tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14426) RBF: Add delegation token total count as one of the federation metrics
[ https://issues.apache.org/jira/browse/HDFS-14426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834183#comment-16834183 ] Fengnan Li edited comment on HDFS-14426 at 5/6/19 8:53 PM: --- [^HDFS-14426-HDFS-13891.005.patch] is exactly the same as [^HDFS-14426-HDFS-13891.004.patch] Just want to trigger the build again. was (Author: fengnanli): [^HDFS-14426-HDFS-13891.003.patch] is exactly the same as [^HDFS-14426-HDFS-13891.003.patch] Just want to trigger the build again. > RBF: Add delegation token total count as one of the federation metrics > -- > > Key: HDFS-14426 > URL: https://issues.apache.org/jira/browse/HDFS-14426 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Attachments: HDFS-14426-HDFS-13891.001.patch, > HDFS-14426-HDFS-13891.002.patch, HDFS-14426-HDFS-13891.003.patch, > HDFS-14426-HDFS-13891.004.patch, HDFS-14426-HDFS-13891.005.patch, > HDFS-14426.001.patch > > > Currently router doesn't report the total number of current valid delegation > tokens it has, but this piece of information is useful for monitoring and > understanding the real time situation of tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14426) RBF: Add delegation token total count as one of the federation metrics
[ https://issues.apache.org/jira/browse/HDFS-14426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fengnan Li updated HDFS-14426: -- Attachment: HDFS-14426-HDFS-13891.005.patch > RBF: Add delegation token total count as one of the federation metrics > -- > > Key: HDFS-14426 > URL: https://issues.apache.org/jira/browse/HDFS-14426 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Attachments: HDFS-14426-HDFS-13891.001.patch, > HDFS-14426-HDFS-13891.002.patch, HDFS-14426-HDFS-13891.003.patch, > HDFS-14426-HDFS-13891.004.patch, HDFS-14426-HDFS-13891.005.patch, > HDFS-14426.001.patch > > > Currently router doesn't report the total number of current valid delegation > tokens it has, but this piece of information is useful for monitoring and > understanding the real time situation of tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834180#comment-16834180
]
Hadoop QA commented on HDFS-14390:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14390 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12967960/HDFS-14390.006.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 824ac84dd005 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 12b7059 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26754/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26754/testReport/ |
| Max. process+thread count | 3015 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26754/console |
| Powered by |
[jira] [Comment Edited] (HDFS-13955) RBF: Support secure Namenode in NamenodeHeartbeatService
[ https://issues.apache.org/jira/browse/HDFS-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834174#comment-16834174 ] CR Hota edited comment on HDFS-13955 at 5/6/19 8:45 PM: [~brahmareddy] [~elgoiri] Sadly am yet to see the change for HDFS-14460 in router branch. Can you help coordinate with Infra? was (Author: crh): [~brahmareddy] [~elgoiri] Sadly am yet to see the change for HDFS-14460 yet in router branch. Can you help coordinate with Infra? > RBF: Support secure Namenode in NamenodeHeartbeatService > > > Key: HDFS-13955 > URL: https://issues.apache.org/jira/browse/HDFS-13955 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: HDFS-13955-HDFS-13532.000.patch, > HDFS-13955-HDFS-13532.001.patch > > > Currently, the NamenodeHeartbeatService uses JMX to get the metrics from the > Namenodes. We should support HTTPs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13955) RBF: Support secure Namenode in NamenodeHeartbeatService
[ https://issues.apache.org/jira/browse/HDFS-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834174#comment-16834174 ] CR Hota commented on HDFS-13955: [~brahmareddy] [~elgoiri] Sadly am yet to see the change for HDFS-14460 yet in router branch. Can you help coordinate with Infra? > RBF: Support secure Namenode in NamenodeHeartbeatService > > > Key: HDFS-13955 > URL: https://issues.apache.org/jira/browse/HDFS-13955 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: HDFS-13955-HDFS-13532.000.patch, > HDFS-13955-HDFS-13532.001.patch > > > Currently, the NamenodeHeartbeatService uses JMX to get the metrics from the > Namenodes. We should support HTTPs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13677) Dynamic refresh Disk configuration results in overwriting VolumeMap
[ https://issues.apache.org/jira/browse/HDFS-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-13677: - Target Version/s: (was: 2.9.0, 2.10.0, 2.8.6) Fix Version/s: 2.9.3 2.8.6 2.10.0 Thanks for the branch-2 patch [~sodonnell]. +1, I have committed after compiling and running affected unit tests locally. > Dynamic refresh Disk configuration results in overwriting VolumeMap > --- > > Key: HDFS-13677 > URL: https://issues.apache.org/jira/browse/HDFS-13677 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: xuzq >Assignee: xuzq >Priority: Blocker > Fix For: 2.10.0, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3 > > Attachments: HDFS-13677-001.patch, HDFS-13677-002-2.9-branch.patch, > HDFS-13677-002.patch, image-2018-06-14-13-05-54-354.png, > image-2018-06-14-13-10-24-032.png > > > When I added a new disk by dynamically refreshing the configuration, an > exception "FileNotFound while finding block" was caused. > > The steps are as follows: > 1.Change the hdfs-site.xml of DataNode to add a new disk. > 2.Refresh the configuration by "./bin/hdfs dfsadmin -reconfig datanode > :50020 start" > > The error is like: > ``` > VolumeScannerThread(/media/disk5/hdfs/dn): FileNotFound while finding block > BP-233501496-*.*.*.*-1514185698256:blk_1620868560_547245090 on volume > /media/disk5/hdfs/dn > org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not > found for BP-1997955181-*.*.*.*-1514186468560:blk_1090885868_17145082 > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.getReplica(BlockSender.java:471) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:240) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:553) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:148) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:254) > at java.lang.Thread.run(Thread.java:748) > ``` > I added some logs for confirmation, as follows: > Log Code like: > !image-2018-06-14-13-05-54-354.png! > And the result is like: > !image-2018-06-14-13-10-24-032.png! > The Size of 'VolumeMap' has been reduced, and We found the 'VolumeMap' to be > overridden by the new Disk Block by the method 'ReplicaMap.addAll(ReplicaMap > other)'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDDS-1495) Create hadoop/ozone docker images with inline build process
[
https://issues.apache.org/jira/browse/HDDS-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang moved HADOOP-16091 to HDDS-1495:
--
Workflow: patch-available, re-open possible (was: no-reopen-closed,
patch-avail)
Key: HDDS-1495 (was: HADOOP-16091)
Project: Hadoop Distributed Data Store (was: Hadoop Common)
> Create hadoop/ozone docker images with inline build process
> ---
>
> Key: HDDS-1495
> URL: https://issues.apache.org/jira/browse/HDDS-1495
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Eric Yang
>Priority: Major
> Attachments: HADOOP-16091.001.patch, HADOOP-16091.002.patch
>
>
> This is proposed by [~eyang] in
> [this|https://lists.apache.org/thread.html/33ac54bdeacb4beb023ebd452464603aaffa095bd104cb43c22f484e@%3Chdfs-dev.hadoop.apache.org%3E]
> mailing thread.
> {quote}1, 3. There are 38 Apache projects hosting docker images on Docker hub
> using Apache Organization. By browsing Apache github mirror. There are only 7
> projects using a separate repository for docker image build. Popular projects
> official images are not from Apache organization, such as zookeeper, tomcat,
> httpd. We may not disrupt what other Apache projects are doing, but it looks
> like inline build process is widely employed by majority of projects such as
> Nifi, Brooklyn, thrift, karaf, syncope and others. The situation seems a bit
> chaotic for Apache as a whole. However, Hadoop community can decide what is
> best for Hadoop. My preference is to remove ozone from source tree naming, if
> Ozone is intended to be subproject of Hadoop for long period of time. This
> enables Hadoop community to host docker images for various subproject without
> having to check out several source tree to trigger a grand build. However,
> inline build process seems more popular than separated process. Hence, I
> highly recommend making docker build inline if possible.
> {quote}
> The main challenges are also discussed in the thread:
> {code:java}
> 3. Technically it would be possible to add the Dockerfile to the source
> tree and publish the docker image together with the release by the
> release manager but it's also problematic:
> {code}
> a) there is no easy way to stage the images for the vote
> c) it couldn't be flagged as automated on dockerhub
> d) It couldn't support the critical updates.
> * Updating existing images (for example in case of an ssl bug, rebuild
> all the existing images with exactly the same payload but updated base
> image/os environment)
> * Creating image for older releases (We would like to provide images,
> for hadoop 2.6/2.7/2.7/2.8/2.9. Especially for doing automatic testing
> with different versions).
> {code:java}
> {code}
> The a) can be solved (as [~eyang] suggested) with using a personal docker
> image during the vote and publish it to the dockerhub after the vote (in case
> the permission can be set by the INFRA)
> Note: based on LEGAL-270 and linked discussion both approaches (inline build
> process / external build process) are compatible with the apache release.
> Note: HDDS-851 and HADOOP-14898 contains more information about these
> problems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834164#comment-16834164
]
Eric Yang commented on HDDS-1458:
-
{quote}Not exactly, I am talking about a differnt thing. Let's imagine that you
start two build paralell. How do you know which image is used to execute the
tests? You can't be sure. There is no direct relation ship between checked out
source code, docker image which is created and the compose files. (We need
build specific tag names for that which are saved to somewhere).{quote}
This can be done by using:
{code}
mvn -Dversion=[customized_version]
{code}
This property will be materialized via maven resource plugin of Dockerfile with
customized version. Hence, each build generates a unique version number that
is identical between jar, tar, and docker image. Post build will need to have
a clean up script to delete old Docker images. I think this may have already
been done in most projects on Jenkins.
{quote}BTW, the original question (how the blockade/fault injection tests can
be executed as part of the maven build) can be solved easily (IMHO). I uploaded
the patch to demonstrate it.{quote}
I don't see your patch in this issue. Where did you upload the patch?
>From today's Ozone meeting, we agree on making sure that fault injection test
>can work with docker image that carry ozone binaries, and docker image that
>mount ozone binaries. This ensure that the current work flow is not
>disrupted. I will update patch to reflect this agreement.
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HDDS-1458.001.patch, HDDS-1458.002.patch,
> HDDS-1458.003.patch
>
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13995) RBF: Security documentation
[ https://issues.apache.org/jira/browse/HDFS-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834160#comment-16834160 ] CR Hota commented on HDFS-13995: [~brahmareddy] [~surendrasingh] [~ajisakaa] Gentle reminder ! > RBF: Security documentation > --- > > Key: HDFS-13995 > URL: https://issues.apache.org/jira/browse/HDFS-13995 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-13995-HDFS-13891.001.patch, > HDFS-13995-HDFS-13891.002.patch, HDFS-13995-HDFS-13891.003.patch > > > Documentation for users under the section relating to security needs to be > updated once security work is completely. > [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html#Security] > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14475) RBF: Expose router security enabled status on the UI
[ https://issues.apache.org/jira/browse/HDFS-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834158#comment-16834158 ] CR Hota commented on HDFS-14475: [~elgoiri] Can you help commit this? This is the same patch that was reviewed by you and [~brahmareddy] in HDFS-12510. > RBF: Expose router security enabled status on the UI > > > Key: HDFS-14475 > URL: https://issues.apache.org/jira/browse/HDFS-14475 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-14475-HDFS-13891.001.patch > > > This is a branched off Jira to expose metric so that router's security status > can be displayed on the UI. We are still unclear if more work needs to be > done for dealing with CORS etc. > https://issues.apache.org/jira/browse/HDFS-12510 will continue to track that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[ https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834151#comment-16834151 ] Chen Liang commented on HDFS-12979: --- Sorry I missed [~xkrogen]'s previous comment when posting the patch (should've refreshed the page). This is one alternative I thought of as well. I was mainly concerned about the complexity of having multiple primary checker and I am not sure yet if there is any other place that is based on the assumption of one primary checker. But yeah, this does worth considering, looking into this. > StandbyNode should upload FsImage to ObserverNode after checkpointing. > -- > > Key: HDFS-12979 > URL: https://issues.apache.org/jira/browse/HDFS-12979 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Konstantin Shvachko >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-12979.001.patch, HDFS-12979.002.patch > > > ObserverNode does not create checkpoints. So it's fsimage file can get very > old making bootstrap of ObserverNode too long. A StandbyNode should copy > latest fsimage to ObserverNode(s) along with ANN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1424) Support multi-container robot test execution
[
https://issues.apache.org/jira/browse/HDDS-1424?focusedWorklogId=238079=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238079
]
ASF GitHub Bot logged work on HDDS-1424:
Author: ASF GitHub Bot
Created on: 06/May/19 19:50
Start Date: 06/May/19 19:50
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on issue #726: HDDS-1424. Support
multi-container robot test execution
URL: https://github.com/apache/hadoop/pull/726#issuecomment-489751889
+1 from me too. This will allow more acceptance tests to be added easily.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 238079)
Time Spent: 2h 40m (was: 2.5h)
> Support multi-container robot test execution
>
>
> Key: HDDS-1424
> URL: https://issues.apache.org/jira/browse/HDDS-1424
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> The ./smoketest folder in the distribution package contains robotframework
> based test scripts to test the main behaviour of Ozone.
> The tests have two layers:
> 1. robot test definitions to execute commands and assert the results (on a
> given host machine)
> 2. ./smoketest/test.sh which starts/stops the docker-compose based
> environments AND execute the selected robot tests inside the right hosts
> The second one (test.sh) has some serious limitations:
> 1. all the tests are executed inside the same container (om):
> https://github.com/apache/hadoop/blob/5f951ea2e39ae4dfe554942baeec05849cd7d3c2/hadoop-ozone/dist/src/main/smoketest/test.sh#L89
> Some of the tests (ozonesecure-mr, ozonefs) may require the flexibility to
> execute different robot tests in different containers.
> 2. The definition of the global test set is complex and hard to understood.
> The current code is:
> {code}
>TESTS=("basic")
>execute_tests ozone "${TESTS[@]}"
>TESTS=("auditparser")
>execute_tests ozone "${TESTS[@]}"
>TESTS=("ozonefs")
>execute_tests ozonefs "${TESTS[@]}"
>TESTS=("basic")
>execute_tests ozone-hdfs "${TESTS[@]}"
>TESTS=("s3")
>execute_tests ozones3 "${TESTS[@]}"
>TESTS=("security")
>execute_tests ozonesecure .
> {code}
> For example for ozonesecure the TESTS is not used. And the usage of bash
> lists require additional complexity in the execute_tests function.
> I propose here a very lightweight refactor. Instead of including both the
> test definitions AND the helper methods in test.sh I would separate them.
> Let's put a test.sh to each of the compose directories. The separated test.sh
> can include common methods from a main shell script. For example:
> {code}
> source "$COMPOSE_DIR/../testlib.sh"
> start_docker_env
> execute_robot_test scm basic/basic.robot
> execute_robot_test scm s3
> stop_docker_env
> generate_report
> {code}
> This is a more clean and more flexible definition. It's easy to execute just
> this test (as it's saved to the compose/ozones3 directory. And it's more
> flexible.
> Other example, where multiple containers are used to execute tests:
> {code}
> source "$COMPOSE_DIR/../testlib.sh"
> start_docker_env
> execute_robot_test scm ozonefs/ozonefs.robot
> export OZONE_HOME=/opt/ozone
> execute_robot_test hadoop32 ozonefs/hadoopo3fs.robot
> execute_robot_test hadoop31 ozonefs/hadoopo3fs.robot
> stop_docker_env
> generate_report
> {code}
> With this separation the definition of the helper methods (eg.
> execute_robot_test or stop_docker_env) would also be simplified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14245) Class cast error in GetGroups with ObserverReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834140#comment-16834140
]
Hadoop QA commented on HDFS-14245:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 56s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
13s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}191m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14245 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12967947/HDFS-14245.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a3196c2dc331 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 12b7059 |
| maven | version: Apache Maven 3.3.9 |
|
[jira] [Work logged] (HDDS-1424) Support multi-container robot test execution
[
https://issues.apache.org/jira/browse/HDDS-1424?focusedWorklogId=238032=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238032
]
ASF GitHub Bot logged work on HDDS-1424:
Author: ASF GitHub Bot
Created on: 06/May/19 19:22
Start Date: 06/May/19 19:22
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #726: HDDS-1424. Support
multi-container robot test execution
URL: https://github.com/apache/hadoop/pull/726#discussion_r281307008
##
File path: hadoop-ozone/dist/src/main/compose/ozonefs/test.sh
##
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+export COMPOSE_DIR
+
+# shellcheck source=/dev/null
+source "$COMPOSE_DIR/../testlib.sh"
+
+start_docker_env
+
+execute_robot_test scm ozonefs/ozonefs.robot
+
+
+## TODO: As of the hhe o3fs tests are unstable.
Review comment:
Minor: typo.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 238032)
Time Spent: 2h 10m (was: 2h)
> Support multi-container robot test execution
>
>
> Key: HDDS-1424
> URL: https://issues.apache.org/jira/browse/HDDS-1424
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The ./smoketest folder in the distribution package contains robotframework
> based test scripts to test the main behaviour of Ozone.
> The tests have two layers:
> 1. robot test definitions to execute commands and assert the results (on a
> given host machine)
> 2. ./smoketest/test.sh which starts/stops the docker-compose based
> environments AND execute the selected robot tests inside the right hosts
> The second one (test.sh) has some serious limitations:
> 1. all the tests are executed inside the same container (om):
> https://github.com/apache/hadoop/blob/5f951ea2e39ae4dfe554942baeec05849cd7d3c2/hadoop-ozone/dist/src/main/smoketest/test.sh#L89
> Some of the tests (ozonesecure-mr, ozonefs) may require the flexibility to
> execute different robot tests in different containers.
> 2. The definition of the global test set is complex and hard to understood.
> The current code is:
> {code}
>TESTS=("basic")
>execute_tests ozone "${TESTS[@]}"
>TESTS=("auditparser")
>execute_tests ozone "${TESTS[@]}"
>TESTS=("ozonefs")
>execute_tests ozonefs "${TESTS[@]}"
>TESTS=("basic")
>execute_tests ozone-hdfs "${TESTS[@]}"
>TESTS=("s3")
>execute_tests ozones3 "${TESTS[@]}"
>TESTS=("security")
>execute_tests ozonesecure .
> {code}
> For example for ozonesecure the TESTS is not used. And the usage of bash
> lists require additional complexity in the execute_tests function.
> I propose here a very lightweight refactor. Instead of including both the
> test definitions AND the helper methods in test.sh I would separate them.
> Let's put a test.sh to each of the compose directories. The separated test.sh
> can include common methods from a main shell script. For example:
> {code}
> source "$COMPOSE_DIR/../testlib.sh"
> start_docker_env
> execute_robot_test scm basic/basic.robot
> execute_robot_test scm s3
> stop_docker_env
> generate_report
> {code}
> This is a more clean and more flexible definition. It's easy to execute just
> this test (as it's saved to the compose/ozones3 directory. And it's more
> flexible.
> Other example, where multiple containers are used to execute tests:
> {code}
> source "$COMPOSE_DIR/../testlib.sh"
> start_docker_env
>
[jira] [Work logged] (HDDS-1424) Support multi-container robot test execution
[
https://issues.apache.org/jira/browse/HDDS-1424?focusedWorklogId=238033=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238033
]
ASF GitHub Bot logged work on HDDS-1424:
Author: ASF GitHub Bot
Created on: 06/May/19 19:22
Start Date: 06/May/19 19:22
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #726: HDDS-1424. Support
multi-container robot test execution
URL: https://github.com/apache/hadoop/pull/726#discussion_r281307184
##
File path: hadoop-ozone/dist/src/main/compose/ozones3/test.sh
##
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+export COMPOSE_DIR
+
+# shellcheck source=/dev/null
+source "$COMPOSE_DIR/../testlib.sh"
+
+start_docker_env
+
+execute_robot_test scm basic/basic.robot
Review comment:
Is it deliberate to rerun the basic test within each sub-test?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 238033)
Time Spent: 2h 20m (was: 2h 10m)
> Support multi-container robot test execution
>
>
> Key: HDDS-1424
> URL: https://issues.apache.org/jira/browse/HDDS-1424
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The ./smoketest folder in the distribution package contains robotframework
> based test scripts to test the main behaviour of Ozone.
> The tests have two layers:
> 1. robot test definitions to execute commands and assert the results (on a
> given host machine)
> 2. ./smoketest/test.sh which starts/stops the docker-compose based
> environments AND execute the selected robot tests inside the right hosts
> The second one (test.sh) has some serious limitations:
> 1. all the tests are executed inside the same container (om):
> https://github.com/apache/hadoop/blob/5f951ea2e39ae4dfe554942baeec05849cd7d3c2/hadoop-ozone/dist/src/main/smoketest/test.sh#L89
> Some of the tests (ozonesecure-mr, ozonefs) may require the flexibility to
> execute different robot tests in different containers.
> 2. The definition of the global test set is complex and hard to understood.
> The current code is:
> {code}
>TESTS=("basic")
>execute_tests ozone "${TESTS[@]}"
>TESTS=("auditparser")
>execute_tests ozone "${TESTS[@]}"
>TESTS=("ozonefs")
>execute_tests ozonefs "${TESTS[@]}"
>TESTS=("basic")
>execute_tests ozone-hdfs "${TESTS[@]}"
>TESTS=("s3")
>execute_tests ozones3 "${TESTS[@]}"
>TESTS=("security")
>execute_tests ozonesecure .
> {code}
> For example for ozonesecure the TESTS is not used. And the usage of bash
> lists require additional complexity in the execute_tests function.
> I propose here a very lightweight refactor. Instead of including both the
> test definitions AND the helper methods in test.sh I would separate them.
> Let's put a test.sh to each of the compose directories. The separated test.sh
> can include common methods from a main shell script. For example:
> {code}
> source "$COMPOSE_DIR/../testlib.sh"
> start_docker_env
> execute_robot_test scm basic/basic.robot
> execute_robot_test scm s3
> stop_docker_env
> generate_report
> {code}
> This is a more clean and more flexible definition. It's easy to execute just
> this test (as it's saved to the compose/ozones3 directory. And it's more
> flexible.
> Other example, where multiple containers are used to execute tests:
> {code}
> source "$COMPOSE_DIR/../testlib.sh"
> start_docker_env
> execute_robot_test
[jira] [Work logged] (HDDS-1424) Support multi-container robot test execution
[
https://issues.apache.org/jira/browse/HDDS-1424?focusedWorklogId=238034=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238034
]
ASF GitHub Bot logged work on HDDS-1424:
Author: ASF GitHub Bot
Created on: 06/May/19 19:24
Start Date: 06/May/19 19:24
Worklog Time Spent: 10m
Work Description: arp7 commented on issue #726: HDDS-1424. Support
multi-container robot test execution
URL: https://github.com/apache/hadoop/pull/726#issuecomment-489743634
/retest
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 238034)
Time Spent: 2.5h (was: 2h 20m)
> Support multi-container robot test execution
>
>
> Key: HDDS-1424
> URL: https://issues.apache.org/jira/browse/HDDS-1424
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> The ./smoketest folder in the distribution package contains robotframework
> based test scripts to test the main behaviour of Ozone.
> The tests have two layers:
> 1. robot test definitions to execute commands and assert the results (on a
> given host machine)
> 2. ./smoketest/test.sh which starts/stops the docker-compose based
> environments AND execute the selected robot tests inside the right hosts
> The second one (test.sh) has some serious limitations:
> 1. all the tests are executed inside the same container (om):
> https://github.com/apache/hadoop/blob/5f951ea2e39ae4dfe554942baeec05849cd7d3c2/hadoop-ozone/dist/src/main/smoketest/test.sh#L89
> Some of the tests (ozonesecure-mr, ozonefs) may require the flexibility to
> execute different robot tests in different containers.
> 2. The definition of the global test set is complex and hard to understood.
> The current code is:
> {code}
>TESTS=("basic")
>execute_tests ozone "${TESTS[@]}"
>TESTS=("auditparser")
>execute_tests ozone "${TESTS[@]}"
>TESTS=("ozonefs")
>execute_tests ozonefs "${TESTS[@]}"
>TESTS=("basic")
>execute_tests ozone-hdfs "${TESTS[@]}"
>TESTS=("s3")
>execute_tests ozones3 "${TESTS[@]}"
>TESTS=("security")
>execute_tests ozonesecure .
> {code}
> For example for ozonesecure the TESTS is not used. And the usage of bash
> lists require additional complexity in the execute_tests function.
> I propose here a very lightweight refactor. Instead of including both the
> test definitions AND the helper methods in test.sh I would separate them.
> Let's put a test.sh to each of the compose directories. The separated test.sh
> can include common methods from a main shell script. For example:
> {code}
> source "$COMPOSE_DIR/../testlib.sh"
> start_docker_env
> execute_robot_test scm basic/basic.robot
> execute_robot_test scm s3
> stop_docker_env
> generate_report
> {code}
> This is a more clean and more flexible definition. It's easy to execute just
> this test (as it's saved to the compose/ozones3 directory. And it's more
> flexible.
> Other example, where multiple containers are used to execute tests:
> {code}
> source "$COMPOSE_DIR/../testlib.sh"
> start_docker_env
> execute_robot_test scm ozonefs/ozonefs.robot
> export OZONE_HOME=/opt/ozone
> execute_robot_test hadoop32 ozonefs/hadoopo3fs.robot
> execute_robot_test hadoop31 ozonefs/hadoopo3fs.robot
> stop_docker_env
> generate_report
> {code}
> With this separation the definition of the helper methods (eg.
> execute_robot_test or stop_docker_env) would also be simplified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14475) RBF: Expose router security enabled status on the UI
[
https://issues.apache.org/jira/browse/HDFS-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834127#comment-16834127
]
Hadoop QA commented on HDFS-14475:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
11s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 22m
29s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14475 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12967959/HDFS-14475-HDFS-13891.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b3074dfb77c1 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / 206b082 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26755/testReport/ |
| Max. process+thread count | 1473 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26755/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Expose router security enabled status on the UI
>
>
>
[jira] [Updated] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[ https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12979: -- Status: Patch Available (was: Open) > StandbyNode should upload FsImage to ObserverNode after checkpointing. > -- > > Key: HDFS-12979 > URL: https://issues.apache.org/jira/browse/HDFS-12979 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Konstantin Shvachko >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-12979.001.patch, HDFS-12979.002.patch > > > ObserverNode does not create checkpoints. So it's fsimage file can get very > old making bootstrap of ObserverNode too long. A StandbyNode should copy > latest fsimage to ObserverNode(s) along with ANN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[
https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834109#comment-16834109
]
Chen Liang commented on HDFS-12979:
---
Post v002 patch. Added a new http code \{{SC_ACCEPTED}} to indicated successful
upload to observer. Open to suggestions for alternatives.
> StandbyNode should upload FsImage to ObserverNode after checkpointing.
> --
>
> Key: HDFS-12979
> URL: https://issues.apache.org/jira/browse/HDFS-12979
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Konstantin Shvachko
>Assignee: Chen Liang
>Priority: Major
> Attachments: HDFS-12979.001.patch, HDFS-12979.002.patch
>
>
> ObserverNode does not create checkpoints. So it's fsimage file can get very
> old making bootstrap of ObserverNode too long. A StandbyNode should copy
> latest fsimage to ObserverNode(s) along with ANN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[ https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12979: -- Attachment: HDFS-12979.002.patch > StandbyNode should upload FsImage to ObserverNode after checkpointing. > -- > > Key: HDFS-12979 > URL: https://issues.apache.org/jira/browse/HDFS-12979 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Konstantin Shvachko >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-12979.001.patch, HDFS-12979.002.patch > > > ObserverNode does not create checkpoints. So it's fsimage file can get very > old making bootstrap of ObserverNode too long. A StandbyNode should copy > latest fsimage to ObserverNode(s) along with ANN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1384) TestBlockOutputStreamWithFailures is failing
[
https://issues.apache.org/jira/browse/HDDS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834102#comment-16834102
]
Eric Yang commented on HDDS-1384:
-
Maybe the better way to fix race condition of port binding is to ensure the
ephemeral port range is 1 and above, where dynamic ports binding is in
separated range from ephemeral port.
Ephemeral ports
{code}
sudo sysctl -w net.ipv4.ip_local_port_range="1 65535"
{code}
Dynamic ports
{code}
sudo sysctl -w net.ipv4.ip_local_reserved_ports="6000, 9000"
{code}
Keep in mind that reserved ports are specified in number instead of ranges.
> TestBlockOutputStreamWithFailures is failing
>
>
> Key: HDDS-1384
> URL: https://issues.apache.org/jira/browse/HDDS-1384
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> TestBlockOutputStreamWithFailures is failing with the following error
> {noformat}
> 2019-04-04 18:52:43,240 INFO volume.ThrottledAsyncChecker
> (ThrottledAsyncChecker.java:schedule(140)) - Scheduling a check for
> org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,240 INFO volume.HddsVolumeChecker
> (HddsVolumeChecker.java:checkAllVolumes(203)) - Scheduled health check for
> volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,241 ERROR server.GrpcService
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1: Failed to
> start Grpc server
> java.io.IOException: Failed to bind
> at
> org.apache.ratis.thirdparty.io.grpc.netty.NettyServer.start(NettyServer.java:253)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:166)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:81)
> at org.apache.ratis.grpc.server.GrpcService.startImpl(GrpcService.java:144)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerRpcWithProxy.start(RaftServerRpcWithProxy.java:69)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$start$3(RaftServerProxy.java:300)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:298)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:419)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:186)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.start(DatanodeStateMachine.java:169)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$startDaemon$0(DatanodeStateMachine.java:338)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at
> org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
> at
> org.apache.ratis.thirdparty.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
> at
> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> at
>
[jira] [Commented] (HDDS-1384) TestBlockOutputStreamWithFailures is failing
[
https://issues.apache.org/jira/browse/HDDS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834101#comment-16834101
]
Hudson commented on HDDS-1384:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16507 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16507/])
Revert "HDDS-1384. TestBlockOutputStreamWithFailures is failing" (elek: rev
fb7c1cad0ea93406a7272872c888d06e4e56620a)
* (edit)
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java
> TestBlockOutputStreamWithFailures is failing
>
>
> Key: HDDS-1384
> URL: https://issues.apache.org/jira/browse/HDDS-1384
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> TestBlockOutputStreamWithFailures is failing with the following error
> {noformat}
> 2019-04-04 18:52:43,240 INFO volume.ThrottledAsyncChecker
> (ThrottledAsyncChecker.java:schedule(140)) - Scheduling a check for
> org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,240 INFO volume.HddsVolumeChecker
> (HddsVolumeChecker.java:checkAllVolumes(203)) - Scheduled health check for
> volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,241 ERROR server.GrpcService
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1: Failed to
> start Grpc server
> java.io.IOException: Failed to bind
> at
> org.apache.ratis.thirdparty.io.grpc.netty.NettyServer.start(NettyServer.java:253)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:166)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:81)
> at org.apache.ratis.grpc.server.GrpcService.startImpl(GrpcService.java:144)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerRpcWithProxy.start(RaftServerRpcWithProxy.java:69)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$start$3(RaftServerProxy.java:300)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:298)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:419)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:186)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.start(DatanodeStateMachine.java:169)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$startDaemon$0(DatanodeStateMachine.java:338)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at
> org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
> at
> org.apache.ratis.thirdparty.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
> at
> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>
[jira] [Comment Edited] (HDDS-1451) SCMBlockManager findPipeline and createPipeline are not lock protected
[
https://issues.apache.org/jira/browse/HDDS-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831983#comment-16831983
]
Aravindan Vijayan edited comment on HDDS-1451 at 5/6/19 6:27 PM:
-
[~msingh] The getPipelines() and createPipeline() already seem to have a lock
in their implementation. However, the problem described here involves a race
condition between the call to getPipelines and createPipelines in
BlockManagerImpl#allocateBlock. Is my understanding correct? Do you see any
other approach to this?
was (Author: avijayan):
[~msingh] The getPipelines() and createPipeline() already seem to have a lock
in their implementation. However, the problem describe involves a race
condition between the call to getPipelines and createPipelines in
BlockManagerImpl#allocateBlock. Is my understanding correct? Do you see any
other approach to this?
> SCMBlockManager findPipeline and createPipeline are not lock protected
> --
>
> Key: HDDS-1451
> URL: https://issues.apache.org/jira/browse/HDDS-1451
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: MiniOzoneChaosCluster
>
> SCM BlockManager may try to allocate pipelines in the cases when it is not
> needed. This happens because BlockManagerImpl#allocateBlock is not lock
> protected, so multiple pipelines can be allocated from it. One of the
> pipeline allocation can fail even when one of the existing pipeline already
> exists.
> {code}
> 2019-04-22 22:34:14,336 INFO pipeline.RatisPipelineProvider
> (RatisPipelineProvider.java:lambda$create$1(103)) - pipeline Pipeline[ Id:
> 6f4bb2d7-d660-4f9f-bc06-72b10f9a738e, Nodes: 76e1a493-fd55-4d67-9f5
> 5-c04fd6bd3a33{ip: 192.168.0.104, host: 192.168.0.104, certSerialId:
> null}2b9850b2-aed3-4a40-91b5-2447dc5246bf{ip: 192.168.0.104, host:
> 192.168.0.104, certSerialId: null}12248721-ea6a-453f-8dad-fc7fbe692f
> d2{ip: 192.168.0.104, host: 192.168.0.104, certSerialId: null}, Type:RATIS,
> Factor:THREE, State:OPEN]
> 2019-04-22 22:34:14,386 INFO impl.RoleInfo
> (RoleInfo.java:shutdownLeaderElection(134)) -
> e17b7852-4691-40c7-8791-ad0b0da5201f: shutdown LeaderElection
> 2019-04-22 22:34:14,388 INFO pipeline.RatisPipelineProvider
> (RatisPipelineProvider.java:lambda$create$1(103)) - pipeline Pipeline[ Id:
> 552e28f3-98d9-41f3-86e0-c1b9494838a5, Nodes: e17b7852-4691-40c7-879
> 1-ad0b0da5201f{ip: 192.168.0.104, host: 192.168.0.104, certSerialId:
> null}fd365bac-e26e-4b11-afd8-9d08cd1b0521{ip: 192.168.0.104, host:
> 192.168.0.104, certSerialId: null}9583a007-7f02-4074-9e26-19bc18e29e
> c5{ip: 192.168.0.104, host: 192.168.0.104, certSerialId: null}, Type:RATIS,
> Factor:THREE, State:OPEN]
> 2019-04-22 22:34:14,388 INFO impl.RoleInfo (RoleInfo.java:updateAndGet(143))
> - e17b7852-4691-40c7-8791-ad0b0da5201f: start FollowerState
> 2019-04-22 22:34:14,388 INFO pipeline.RatisPipelineProvider
> (RatisPipelineProvider.java:lambda$create$1(103)) - pipeline Pipeline[ Id:
> 5383151b-d625-4362-a7dd-c0d353acaf76, Nodes: 80f16ad6-3879-4a64-a3c
> 7-7719813cc139{ip: 192.168.0.104, host: 192.168.0.104, certSerialId:
> null}082ce481-7fb0-4f88-ac21-82609290a6a2{ip: 192.168.0.104, host:
> 192.168.0.104, certSerialId: null}dd5f5a70-0217-4577-b7a2-c42aa139d1
> 8a{ip: 192.168.0.104, host: 192.168.0.104, certSerialId: null}, Type:RATIS,
> Factor:THREE, State:OPEN]
> 2019-04-22 22:34:14,389 INFO pipeline.RatisPipelineProvider
> (RatisPipelineProvider.java:lambda$create$1(103)) - pipeline Pipeline[ Id:
> be4854e5-7933-4caa-b32e-f482cf500247, Nodes: 6e2356f1-479d-498b-876
> a-1c90623c498b{ip: 192.168.0.104, host: 192.168.0.104, certSerialId:
> null}8ac46d94-9975-4eea-9448-2618c69d7bf3{ip: 192.168.0.104, host:
> 192.168.0.104, certSerialId: null}a3ed36a1-44ca-47b2-b9b3-5aeef04595
> 18{ip: 192.168.0.104, host: 192.168.0.104, certSerialId: null}, Type:RATIS,
> Factor:THREE, State:OPEN]
> 2019-04-22 22:34:14,390 INFO pipeline.RatisPipelineProvider
> (RatisPipelineProvider.java:lambda$create$1(103)) - pipeline Pipeline[ Id:
> 21e368e2-f82a-4c61-9cc3-06e8de22ea6b, Nodes:
> 82632040-5754-4122-b187-331879586842{ip: 192.168.0.104, host: 192.168.0.104,
> certSerialId: null}923c8537-b869-4085-adcb-0a9accdcd089{ip: 192.168.0.104,
> host: 192.168.0.104, certSerialId:
> null}c6d790bf-e3a6-4064-acb5-f74796cd38a9{ip: 192.168.0.104, host:
> 192.168.0.104, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN]
> 2019-04-22 22:34:14,390 INFO pipeline.RatisPipelineProvider
> (RatisPipelineProvider.java:lambda$create$1(103)) - pipeline Pipeline[ Id:
> cccbc2ed-e0e2-4578-a8a2-94f4b645be52, Nodes:
>
[jira] [Work logged] (HDDS-1441) Remove usage of getRetryFailureException
[ https://issues.apache.org/jira/browse/HDDS-1441?focusedWorklogId=237987=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-237987 ] ASF GitHub Bot logged work on HDDS-1441: Author: ASF GitHub Bot Created on: 06/May/19 18:27 Start Date: 06/May/19 18:27 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #745: HDDS-1441. Remove usage of getRetryFailureException. (swagle) URL: https://github.com/apache/hadoop/pull/745#issuecomment-489724608 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 28 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 22 | Maven dependency ordering for branch | | +1 | mvninstall | 405 | trunk passed | | +1 | compile | 203 | trunk passed | | +1 | checkstyle | 47 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 796 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 122 | trunk passed | | 0 | spotbugs | 234 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 417 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for patch | | -1 | mvninstall | 38 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 15 | hadoop-ozone in the patch failed. | | -1 | compile | 18 | hadoop-hdds in the patch failed. | | -1 | compile | 16 | hadoop-ozone in the patch failed. | | -1 | javac | 18 | hadoop-hdds in the patch failed. | | -1 | javac | 16 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 17 | The patch fails to run checkstyle in hadoop-hdds | | -0 | checkstyle | 14 | The patch fails to run checkstyle in hadoop-ozone | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 3 | The patch has no ill-formed XML file. | | +1 | shadedclient | 628 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 19 | hadoop-hdds in the patch failed. | | -1 | javadoc | 17 | hadoop-ozone in the patch failed. | | -1 | findbugs | 30 | hadoop-hdds in the patch failed. | | -1 | findbugs | 18 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 24 | hadoop-hdds in the patch failed. | | -1 | unit | 17 | hadoop-ozone in the patch failed. | | +1 | asflicense | 25 | The patch does not generate ASF License warnings. | | | | 3067 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-745/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/745 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 007e5e55bb1d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 12b7059 | | Default Java | 1.8.0_191 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-745/7/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-745/7/artifact/out/patch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-745/7/artifact/out/patch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-745/7/artifact/out/patch-compile-hadoop-ozone.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-745/7/artifact/out/patch-compile-hadoop-hdds.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-745/7/artifact/out/patch-compile-hadoop-ozone.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-745/7/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-745/out/maven-patch-checkstyle-hadoop-hdds.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-745/7/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-745/out/maven-patch-checkstyle-hadoop-ozone.txt | | javadoc |
[jira] [Commented] (HDDS-1384) TestBlockOutputStreamWithFailures is failing
[
https://issues.apache.org/jira/browse/HDDS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834083#comment-16834083
]
Elek, Marton commented on HDDS-1384:
This is reverted as other problems are introduced:
https://builds.apache.org/job/hadoop-multibranch/job/PR-773/3/testReport/
> TestBlockOutputStreamWithFailures is failing
>
>
> Key: HDDS-1384
> URL: https://issues.apache.org/jira/browse/HDDS-1384
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> TestBlockOutputStreamWithFailures is failing with the following error
> {noformat}
> 2019-04-04 18:52:43,240 INFO volume.ThrottledAsyncChecker
> (ThrottledAsyncChecker.java:schedule(140)) - Scheduling a check for
> org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,240 INFO volume.HddsVolumeChecker
> (HddsVolumeChecker.java:checkAllVolumes(203)) - Scheduled health check for
> volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,241 ERROR server.GrpcService
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1: Failed to
> start Grpc server
> java.io.IOException: Failed to bind
> at
> org.apache.ratis.thirdparty.io.grpc.netty.NettyServer.start(NettyServer.java:253)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:166)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:81)
> at org.apache.ratis.grpc.server.GrpcService.startImpl(GrpcService.java:144)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerRpcWithProxy.start(RaftServerRpcWithProxy.java:69)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$start$3(RaftServerProxy.java:300)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:298)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:419)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:186)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.start(DatanodeStateMachine.java:169)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$startDaemon$0(DatanodeStateMachine.java:338)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at
> org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
> at
> org.apache.ratis.thirdparty.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
> at
> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> ... 1 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail:
[jira] [Commented] (HDDS-1384) TestBlockOutputStreamWithFailures is failing
[
https://issues.apache.org/jira/browse/HDDS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834085#comment-16834085
]
Arpit Agarwal commented on HDDS-1384:
-
Thanks for reverting this [~elek]!
> TestBlockOutputStreamWithFailures is failing
>
>
> Key: HDDS-1384
> URL: https://issues.apache.org/jira/browse/HDDS-1384
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> TestBlockOutputStreamWithFailures is failing with the following error
> {noformat}
> 2019-04-04 18:52:43,240 INFO volume.ThrottledAsyncChecker
> (ThrottledAsyncChecker.java:schedule(140)) - Scheduling a check for
> org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,240 INFO volume.HddsVolumeChecker
> (HddsVolumeChecker.java:checkAllVolumes(203)) - Scheduled health check for
> volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,241 ERROR server.GrpcService
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1: Failed to
> start Grpc server
> java.io.IOException: Failed to bind
> at
> org.apache.ratis.thirdparty.io.grpc.netty.NettyServer.start(NettyServer.java:253)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:166)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:81)
> at org.apache.ratis.grpc.server.GrpcService.startImpl(GrpcService.java:144)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerRpcWithProxy.start(RaftServerRpcWithProxy.java:69)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$start$3(RaftServerProxy.java:300)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:298)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:419)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:186)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.start(DatanodeStateMachine.java:169)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$startDaemon$0(DatanodeStateMachine.java:338)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at
> org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
> at
> org.apache.ratis.thirdparty.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
> at
> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> ... 1 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834074#comment-16834074
]
Ashvin commented on HDFS-14390:
---
Thanks [~virajith]. Your suggestion to create util methods for common test code
setup makes sense to me. I have uploaded a new patch [^HDFS-14390.006.patch].
This patch moves kerberos test configuration to {{MiniDFSCluster}} on top of
the previous patch.
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch, HDFS-14390.004.patch, HDFS-14390.005.patch,
> HDFS-14390.006.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashvin updated HDFS-14390:
--
Attachment: HDFS-14390.006.patch
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch, HDFS-14390.004.patch, HDFS-14390.005.patch,
> HDFS-14390.006.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12510) RBF: Add security to UI
[ https://issues.apache.org/jira/browse/HDFS-12510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-12510: --- Status: Open (was: Patch Available) Created a new jira HDFS-14475 to track the work earlier done here. > RBF: Add security to UI > --- > > Key: HDFS-12510 > URL: https://issues.apache.org/jira/browse/HDFS-12510 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Labels: RBF > > HDFS-12273 implemented the UI for Router Based Federation without security. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12510) RBF: Add security to UI
[ https://issues.apache.org/jira/browse/HDFS-12510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-12510: --- Attachment: (was: HDFS-12510-HDFS-13891.001.patch) > RBF: Add security to UI > --- > > Key: HDFS-12510 > URL: https://issues.apache.org/jira/browse/HDFS-12510 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Labels: RBF > > HDFS-12273 implemented the UI for Router Based Federation without security. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[
https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834062#comment-16834062
]
Hadoop QA commented on HDFS-14195:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap |
| | hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality |
| | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14195 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12967943/HDFS-14195.006.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7e2a7adcdc8f 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 12b7059 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26752/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26752/testReport/ |
| Max. process+thread count | 2841 (vs. ulimit of
[jira] [Updated] (HDFS-14475) RBF: Expose router security enabled status on the UI
[ https://issues.apache.org/jira/browse/HDFS-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-14475: --- Status: Patch Available (was: Open) > RBF: Expose router security enabled status on the UI > > > Key: HDFS-14475 > URL: https://issues.apache.org/jira/browse/HDFS-14475 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-14475-HDFS-13891.001.patch > > > This is a branched off Jira to expose metric so that router's security status > can be displayed on the UI. We are still unclear if more work needs to be > done for dealing with CORS etc. > https://issues.apache.org/jira/browse/HDFS-12510 will continue to track that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12510) RBF: Add security to UI
[ https://issues.apache.org/jira/browse/HDFS-12510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834056#comment-16834056 ] CR Hota commented on HDFS-12510: [~elgoiri] [~brahmareddy] Deleted the patch here and leaving all the conversation as is. Created HDFS-14475 to expose the metric and linked these both. > RBF: Add security to UI > --- > > Key: HDFS-12510 > URL: https://issues.apache.org/jira/browse/HDFS-12510 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Labels: RBF > > HDFS-12273 implemented the UI for Router Based Federation without security. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14475) RBF: Expose router security enabled status on the UI
[ https://issues.apache.org/jira/browse/HDFS-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-14475: --- Attachment: HDFS-14475-HDFS-13891.001.patch > RBF: Expose router security enabled status on the UI > > > Key: HDFS-14475 > URL: https://issues.apache.org/jira/browse/HDFS-14475 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-14475-HDFS-13891.001.patch > > > This is a branched off Jira to expose metric so that router's security status > can be displayed on the UI. We are still unclear if more work needs to be > done for dealing with CORS etc. > https://issues.apache.org/jira/browse/HDFS-12510 will continue to track that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1494) Improve logging in client to troubleshoot container not found errors
Aravindan Vijayan created HDDS-1494: --- Summary: Improve logging in client to troubleshoot container not found errors Key: HDDS-1494 URL: https://issues.apache.org/jira/browse/HDDS-1494 Project: Hadoop Distributed Data Store Issue Type: Improvement Affects Versions: 0.4.0 Reporter: Aravindan Vijayan Fix For: 0.5.0 This was seen in HDDS-1485. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1494) Improve logging in client to troubleshoot container not found errors
[
https://issues.apache.org/jira/browse/HDDS-1494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-1494:
Description:
Exception Trace
{code}
19/05/02 09:58:49 ERROR storage.BlockOutputStream: Unexpected Storage Container
Exception:
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
ContainerID 424 does not exist
{code}
This was seen in HDDS-1485.
was:This was seen in HDDS-1485.
> Improve logging in client to troubleshoot container not found errors
>
>
> Key: HDDS-1494
> URL: https://issues.apache.org/jira/browse/HDDS-1494
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Affects Versions: 0.4.0
>Reporter: Aravindan Vijayan
>Priority: Major
> Fix For: 0.5.0
>
>
> Exception Trace
> {code}
> 19/05/02 09:58:49 ERROR storage.BlockOutputStream: Unexpected Storage
> Container Exception:
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> ContainerID 424 does not exist
> {code}
> This was seen in HDDS-1485.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1493) Download and Import Container replicator fails.
[
https://issues.apache.org/jira/browse/HDDS-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey reassigned HDDS-1493:
--
Assignee: Nanda kumar
> Download and Import Container replicator fails.
> ---
>
> Key: HDDS-1493
> URL: https://issues.apache.org/jira/browse/HDDS-1493
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 0.5.0
>Reporter: Aravindan Vijayan
>Assignee: Nanda kumar
>Priority: Major
>
> While running batch jobs (16 threads writing a lot of 10MB+ files), the
> following error is seen in the SCM logs.
> {code}
> ERROR - Can't import the downloaded container data id=317
> {code}
> It is unclear from the logs why this happens. Needs more investigation to
> find the root cause.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[
https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834049#comment-16834049
]
Erik Krogen commented on HDFS-12979:
Good catch, [~vagarychen]. I wonder if we can solve it more simply by saying
that, instead of having one system-wide primary checkpointer, each _receiving_
node has a corresponding primary checkpointer. On the SbNN side, maintain a map
of {{nnAddress -> isPrimaryCheckPointer}}. Each SbNN can be the primary for 0-n
NNs, and each active/observer NN should have exactly 1 primary.
> StandbyNode should upload FsImage to ObserverNode after checkpointing.
> --
>
> Key: HDFS-12979
> URL: https://issues.apache.org/jira/browse/HDFS-12979
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Konstantin Shvachko
>Assignee: Chen Liang
>Priority: Major
> Attachments: HDFS-12979.001.patch
>
>
> ObserverNode does not create checkpoints. So it's fsimage file can get very
> old making bootstrap of ObserverNode too long. A StandbyNode should copy
> latest fsimage to ObserverNode(s) along with ANN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1492) Generated chunk size name too long.
[
https://issues.apache.org/jira/browse/HDDS-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDDS-1492:
---
Priority: Critical (was: Major)
> Generated chunk size name too long.
> ---
>
> Key: HDDS-1492
> URL: https://issues.apache.org/jira/browse/HDDS-1492
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 0.5.0
>Reporter: Aravindan Vijayan
>Priority: Critical
>
> Following exception is seen in SCM logs intermittently.
> {code}
> java.lang.RuntimeException: file name
> 'chunks/2a54b2a153f4a9c5da5f44e2c6f97c60_stream_9c6ac565-e2d4-469c-bd5c-47922a35e798_chunk_10.tmp.2.23115'
> is too long ( > 100 bytes)
> {code}
> We may have to limit the name of the chunk to 100 bytes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1492) Generated chunk size name too long.
[
https://issues.apache.org/jira/browse/HDDS-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDDS-1492:
---
Target Version/s: 0.5.0
> Generated chunk size name too long.
> ---
>
> Key: HDDS-1492
> URL: https://issues.apache.org/jira/browse/HDDS-1492
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 0.5.0
>Reporter: Aravindan Vijayan
>Priority: Critical
>
> Following exception is seen in SCM logs intermittently.
> {code}
> java.lang.RuntimeException: file name
> 'chunks/2a54b2a153f4a9c5da5f44e2c6f97c60_stream_9c6ac565-e2d4-469c-bd5c-47922a35e798_chunk_10.tmp.2.23115'
> is too long ( > 100 bytes)
> {code}
> We may have to limit the name of the chunk to 100 bytes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834031#comment-16834031 ] Wei-Chiu Chuang commented on HDFS-14195: Thanks [~suxingfate] for reporting the issue and submitting the patch. I think the patch looks good overall. The only thing I am concerned is that the patch adds a new field in the delimited output, and there are production tools that parse the delimited oiv output. Additionally, in the future, it is likely we will need to support more fields and more xattr types. I would suggest to either add a command line option to turn this on only when the option is specified, or update the doc stating the delimited output may not be backward compatible, and provide the guideline for tools to adapt. For example, perhaps the tool should instead look at the header to figure out which column is the column for the data it looks for. Thoughts? > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14475) RBF: Expose router security enabled status on the UI
CR Hota created HDFS-14475: -- Summary: RBF: Expose router security enabled status on the UI Key: HDFS-14475 URL: https://issues.apache.org/jira/browse/HDFS-14475 Project: Hadoop HDFS Issue Type: Sub-task Reporter: CR Hota Assignee: CR Hota This is a branched off Jira to expose metric so that router's security status can be displayed on the UI. We are still unclear if more work needs to be done for dealing with CORS etc. https://issues.apache.org/jira/browse/HDFS-12510 will continue to track that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13522) Support observer node from Router-Based Federation
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834021#comment-16834021
]
CR Hota edited comment on HDFS-13522 at 5/6/19 5:28 PM:
[~surendrasingh] Thanks for your comments.
For implementing a queue similar to FairCallQueue, Handlers have to understand
which downstream namenode a call is meant for and also whether its a
read/write. These aspects of a RPC can be inspected only once the call is
handled at the ClientProtocol implementation layer which callqueue won't have
any visibility into. Since we haven't yet finalized how to solve these, in the
interim we have taken a approach as shown in ShortTerm-Routers+Observer.png.
This short term approach segregates reads from writes and helps us serve
latency sensitive engines such as Presto better. We may try to see if this
could become a defacto model where we expose State of Router as read/write just
as Observer's are designed to do. This would solve read/write problem, and
leave us to focus on isolation of writes across various namenodes.
was (Author: crh):
[~surendrasingh] Thanks for your comments.
For implementing a queue similar to FairCallQueue, Handlers have to understand
which downstream namenode a call is meant for and also whether its a
read/write. These aspects of a RPC can be inspected only once the call is
handled at the ClientProtocol implementation layer which callqueue won't have
any visibility into. Since we haven't yet finalized how to solve these, in the
interim we have taken a approach as shown in ShortTerm-Routers+Observer.png.
> Support observer node from Router-Based Federation
> --
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-13522.001.patch, Router+Observer RPC clogging.png,
> ShortTerm-Routers+Observer.png
>
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1489) Unnecessary log messages on console with Ozone shell
[
https://issues.apache.org/jira/browse/HDDS-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Wagle reassigned HDDS-1489:
-
Assignee: Siddharth Wagle
> Unnecessary log messages on console with Ozone shell
> -
>
> Key: HDDS-1489
> URL: https://issues.apache.org/jira/browse/HDDS-1489
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Reporter: Arpit Agarwal
>Assignee: Siddharth Wagle
>Priority: Major
>
> The following log messages are printed on the console when running putkey
> {code}
> $ ozone sh key put /vol1/bucket1/key1 myfile
> 2019-05-03 23:25:15 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=0, SUCCESS, logIndex=1, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c2]
> 2019-05-03 23:25:16 INFO GrpcClientProtocolClient:254 -
> client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140: receive
> RaftClientReply:client-9A5E39BD681D->96c8bede-ba3f-4e01-86d8-53f97957f140@group-8B0913807C4D,
> cid=1, SUCCESS, logIndex=3, commits[96c8bede-ba3f-4e01-86d8-53f97957f140:c4]
> {code}
> These are unnecessary noise and should be suppressed by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[
https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834030#comment-16834030
]
Chen Liang commented on HDFS-12979:
---
Actually another related thing is {{isPrimaryCheckPointer}} boolean check.
Currently this gets set to true when there is just one successful upload. The
idea was that when multiple Standby are uploading, only one should successfully
upload the image and set this to true, the others all have false.
But this is based on the assumption that there is only one image receiver. With
Observer, we have multiple NNs accepting image, meaning there can be multiple
places leading to successful upload. For example, say sbn1 uploads to Active
and set {{isPrimaryCheckPointer}} to true, sbn2 uploads to an Observer and also
set {{isPrimaryCheckPointer}} to true. Then we have two "primary" check pointer.
Seems even if both sbn1 and sbn2 see other NNs in the same order, this is still
an issue . e.g. sbn1 uploads to ANN, succeed, set {{isPrimaryCheckPointer}} to
true, sbn2 in the meantime uploading to ANN failed, but if sbn2 proceeds to
observer before sbn1 (if the loop breaking gets removed), it would still have
the upload succeed and set {{isPrimaryCheckPointer}} to true.
The root cause seems that there is no way to distinguish between successful
upload to ANN and to Observer. Because otherwise we can easily identify that
only the Standby that successfully uploaded to ANN is the primary.
I think the best way might be to introduce another HttpServlet response code in
{{TransferResult, }}that observer returns a different http code on successful
upload to it. And this should not cause issue for rolling upgrade, because we
should only switch SbN to observer when the cluster has already been upgraded,
after which time the new http code gets sent.
Comments are welcome!
> StandbyNode should upload FsImage to ObserverNode after checkpointing.
> --
>
> Key: HDFS-12979
> URL: https://issues.apache.org/jira/browse/HDFS-12979
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Konstantin Shvachko
>Assignee: Chen Liang
>Priority: Major
> Attachments: HDFS-12979.001.patch
>
>
> ObserverNode does not create checkpoints. So it's fsimage file can get very
> old making bootstrap of ObserverNode too long. A StandbyNode should copy
> latest fsimage to ObserverNode(s) along with ANN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13522) Support observer node from Router-Based Federation
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
CR Hota updated HDFS-13522:
---
Attachment: ShortTerm-Routers+Observer.png
> Support observer node from Router-Based Federation
> --
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-13522.001.patch, Router+Observer RPC clogging.png,
> ShortTerm-Routers+Observer.png
>
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
