[jira] [Created] (HDFS-14274) EC: "hdfs dfs -ls -e" throws NPE When EC Policy of Directory set as "-replicate"
Souryakanta Dwivedy created HDFS-14274: -- Summary: EC: "hdfs dfs -ls -e" throws NPE When EC Policy of Directory set as "-replicate" Key: HDFS-14274 URL: https://issues.apache.org/jira/browse/HDFS-14274 Project: Hadoop HDFS Issue Type: Bug Components: erasure-coding Affects Versions: 3.1.1 Reporter: Souryakanta Dwivedy "hdfs dfs -ls -e" throws NPE When EC Policy of Directory set as "-replicate" Steps :- - Create a Directory - Set EC Policy for the Directory as "-replicate" - Check the folder details with command "hdfs dfs -ls -e " Null Pointer exception will display < ls: java.lang.NullPointerException > Actual Result :- ls: java.lang.NullPointerException Expected Result :- Should not throw NullPointerException -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14273) Fix checkstyle issues in BlockLocation's method javadoc
Shweta created HDFS-14273: - Summary: Fix checkstyle issues in BlockLocation's method javadoc Key: HDFS-14273 URL: https://issues.apache.org/jira/browse/HDFS-14273 Project: Hadoop HDFS Issue Type: Bug Reporter: Shweta -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14269) [SBN read] Observer node switches back to Standby after restart
[ https://issues.apache.org/jira/browse/HDFS-14269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14269. Resolution: Won't Fix > [SBN read] Observer node switches back to Standby after restart > --- > > Key: HDFS-14269 > URL: https://issues.apache.org/jira/browse/HDFS-14269 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: Wei-Chiu Chuang >Priority: Major > > The Observer state is not persistent. Once it restarts it becomes Standby > node again. Since it does not participate in NameNode failover for now, > should we assume an Observer node is always an Observer node? This state > should be persisted somewhere, like in Zookeeper. > CC: [~csun] [~shv]. Like to get inputs from you. Have you discussed this > before? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14272) [SBN read] HDFS command line tools does not guarantee consistency
Wei-Chiu Chuang created HDFS-14272:
--
Summary: [SBN read] HDFS command line tools does not guarantee
consistency
Key: HDFS-14272
URL: https://issues.apache.org/jira/browse/HDFS-14272
Project: Hadoop HDFS
Issue Type: Bug
Components: tools
Environment: CDH6.1 (Hadoop 3.0.x) + Consistency Reads from Standby +
SSL + Kerberos + RPC encryption
Reporter: Wei-Chiu Chuang
It is typical for integration tests to create some files and then check its
existence. For example, like the following simple bash script:
{code:java}
# hdfs dfs -touchz /tmp/abc
# hdfs dfs -ls /tmp/abc
{code}
The test execute HDFS bash command sequentially, but it may fail with
Consistent Standby Read because the -ls may not find the file.
Analysis: the second bash command, while launched sequentially after the first
one, is not aware of the state id returned from the first bash command. So
ObserverNode wouldn't wait for the the edits to get propagated, and thus fails.
I've got a cluster where the Observer has tens of seconds of RPC latency, and
this becomes very annoying. (I am still trying to figure out why this Observer
has such a long RPC latency. But that's another story.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14271) [SBN read] StandbyException is logged if Observer is the first NameNode
Wei-Chiu Chuang created HDFS-14271:
--
Summary: [SBN read] StandbyException is logged if Observer is the
first NameNode
Key: HDFS-14271
URL: https://issues.apache.org/jira/browse/HDFS-14271
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs
Affects Versions: 3.3.0
Reporter: Wei-Chiu Chuang
If I transition the first NameNode into Observer state, and then I create a
file from command line, it prints the following StandbyException log message,
as if the command failed. But it actually completed successfully:
{noformat}
[root@weichiu-sbsr-1 ~]# hdfs dfs -touchz /tmp/abf
19/02/12 16:35:17 INFO retry.RetryInvocationHandler:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
Operation category WRITE is not supported in state observer. Visit
https://s.apache.org/sbnn-error
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:98)
at
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1987)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1424)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:762)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:458)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:918)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:853)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2782)
, while invoking $Proxy4.create over
[weichiu-sbsr-1.gce.cloudera.com/172.31.121.145:8020,weichiu-sbsr-2.gce.cloudera.com/172.31.121.140:8020].
Trying to failover immediately.
{noformat}
This is unlike the case when the first NameNode is the Standby, where this
StandbyException is suppressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14270) StateId and TrasactionId not present in Trace level logging
Shweta created HDFS-14270: - Summary: StateId and TrasactionId not present in Trace level logging Key: HDFS-14270 URL: https://issues.apache.org/jira/browse/HDFS-14270 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Shweta While running the command "hdfs --loglevel TRACE dfs -ls /" it was seen that stateId and TransactionId do not appear in the logs. How does one see the stateId and TransactionId in the logs? Is there a different approach? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: DFSClient/DistriburedFileSystem fault injection?
Sergey -you trying to simulate failures client side, or do you have an NN Which actually injects failures all the way up the IPC stack? as if its just client, couldn't registering a fault-injecting client as fs.hdfs.impl could do that FWIW, in the s3a connector we have the "inconsistent" s3 client which mimics some symptoms of delayed consistency; it has a path, a probability of happening and a delay before things become visible. This is in the main hadoop-aws JAR, and is turned on by a configuration switch (yes, it prints a big warning). With a single switch to turn it on, its trivial to enable it in tests On Mon, Feb 11, 2019 at 11:42 PM Sergey Shelukhin wrote: > Hi. > I've been looking for a client-side solution for fault injection in HDFS. > We had a naturally unstable HDFS cluster that helped uncover a lot of > issues in HBase; now that it has been stabilized, we miss it already :) > > To keep testing without actually disrupting others' use of HDFS or having > to deploy a new version, I was thinking about having a client-side schema > (e.g. fhdfs) map to a wrapper over the standard DFS that would inject > failures and delays according to some configs, similar to > https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/FaultInjectFramework.html > > However I wonder if something like this exists already? > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
[jira] [Created] (HDFS-14269) [SBN read] Observer node switches back to Standby after restart
Wei-Chiu Chuang created HDFS-14269: -- Summary: [SBN read] Observer node switches back to Standby after restart Key: HDFS-14269 URL: https://issues.apache.org/jira/browse/HDFS-14269 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.3.0 Reporter: Wei-Chiu Chuang The Observer state is not persistent. Once it restarts it becomes Standby node again. Since it does not participate in NameNode failover for now, should we assume an Observer node is always an Observer node? This state should be persisted somewhere, like in Zookeeper. CC: [~csun] [~shv]. Like to get inputs from you. Have you discussed this before? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1063) Implement OM init in secure cluster
[ https://issues.apache.org/jira/browse/HDDS-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-1063. -- Resolution: Duplicate > Implement OM init in secure cluster > --- > > Key: HDDS-1063 > URL: https://issues.apache.org/jira/browse/HDDS-1063 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > > Implement OM init in secure cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-581) Bootstrap DN with private/public key pair
[ https://issues.apache.org/jira/browse/HDDS-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-581. - Resolution: Duplicate > Bootstrap DN with private/public key pair > - > > Key: HDDS-581 > URL: https://issues.apache.org/jira/browse/HDDS-581 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-581-HDDS-4.00.patch > > > This will create public/private key pair for HDDS datanode if there isn't one > available during secure dn startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: DFSClient/DistriburedFileSystem fault injection?
For what is worth, At Cloudera we have an internal fault injection tool that is based on Chaos Monkey. We use it to kill disks or kill nodes, for example. So it doesn't instrument HDFS directly. On Mon, Feb 11, 2019 at 3:42 PM Sergey Shelukhin wrote: > Hi. > I've been looking for a client-side solution for fault injection in HDFS. > We had a naturally unstable HDFS cluster that helped uncover a lot of > issues in HBase; now that it has been stabilized, we miss it already :) > > To keep testing without actually disrupting others' use of HDFS or having > to deploy a new version, I was thinking about having a client-side schema > (e.g. fhdfs) map to a wrapper over the standard DFS that would inject > failures and delays according to some configs, similar to > https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/FaultInjectFramework.html > > However I wonder if something like this exists already? > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
RE: DFSClient/DistriburedFileSystem fault injection?
Adding the user list :) -Original Message- From: Sergey Shelukhin Sent: Monday, February 11, 2019 3:42 PM To: hdfs-dev@hadoop.apache.org Subject: DFSClient/DistriburedFileSystem fault injection? Hi. I've been looking for a client-side solution for fault injection in HDFS. We had a naturally unstable HDFS cluster that helped uncover a lot of issues in HBase; now that it has been stabilized, we miss it already :) To keep testing without actually disrupting others' use of HDFS or having to deploy a new version, I was thinking about having a client-side schema (e.g. fhdfs) map to a wrapper over the standard DFS that would inject failures and delays according to some configs, similar to https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/FaultInjectFramework.html However I wonder if something like this exists already? - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14268) RBF: Fix the location of the DNs in getDatanodeReport()
Íñigo Goiri created HDFS-14268:
--
Summary: RBF: Fix the location of the DNs in getDatanodeReport()
Key: HDFS-14268
URL: https://issues.apache.org/jira/browse/HDFS-14268
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Íñigo Goiri
Assignee: Íñigo Goiri
When getting all the DNs in the federation, the Router queries each of the
subclusters and aggregates them assigning the subcluster id to the location.
This query uses a {{HashSet}} which provides a "random" order for the results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14267) Add test_libhdfs_ops to libhdfs tests, mark libhdfs_read/write.c as examples
Sahil Takiar created HDFS-14267:
---
Summary: Add test_libhdfs_ops to libhdfs tests, mark
libhdfs_read/write.c as examples
Key: HDFS-14267
URL: https://issues.apache.org/jira/browse/HDFS-14267
Project: Hadoop HDFS
Issue Type: Improvement
Components: libhdfs, native, test
Reporter: Sahil Takiar
{{test_libhdfs_ops.c}} provides test coverage for basic operations against
libhdfs, but currently has to be run manually (e.g. {{mvn install}} does not
run these tests). The goal of this patch is to add {{test_libhdfs_ops.c}} to
the list of tests that are automatically run for libhdfs.
It looks like {{test_libhdfs_ops.c}} was used in conjunction with
{{hadoop-hdfs-project/hadoop-hdfs/src/main/native/tests/test-libhdfs.sh}} to
run some tests against a mini DFS cluster. Now that the
{{NativeMiniDfsCluster}} exists, it makes more sense to use that rather than
rely on an external bash script to start a mini DFS cluster.
The {{libhdfs-tests}} directory (which contains {{test_libhdfs_ops.c}})
contains two other files: {{test_libhdfs_read.c}} and {{test_libhdfs_write.c}}.
At some point, these files might have been used in conjunction with
{{test-libhdfs.sh}} to run some tests manually. However, they (1) largely
overlap with the test coverage provided by {{test_libhdfs_ops.c}} and (2) are
not designed to be run as unit tests. Thus I suggest we move these two files
into a new folder called {{libhdfs-examples}} and use them to further document
how users of libhdfs can use the API. We can move {{test-libhdfs.sh}} into the
examples folder as well given that example files probably require the script to
actually work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1094) Performance testing infrastructure : Special handling for zero-filled chunks on the Datanode
Supratim Deka created HDDS-1094: --- Summary: Performance testing infrastructure : Special handling for zero-filled chunks on the Datanode Key: HDDS-1094 URL: https://issues.apache.org/jira/browse/HDDS-1094 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: Ozone Datanode Reporter: Supratim Deka Goal: Make Ozone chunk Read/Write operations CPU/network bound for specially constructed performance micro benchmarks. Remove disk bandwidth and latency constraints - running ozone data path against extreme low-latency & high throughput storage will expose performance bottlenecks in the flow. But low-latency storage(NVME flash drives, Storage class memory etc) is expensive and availability is limited. Is there a workaround which achieves similar running conditions for the software without actually having the low latency storage? At least for specially constructed datasets - for example zero-filled blocks (*not* zero-length blocks). Required characteristics of the solution: No changes in Ozone client, OM and SCM. Changes limited to Datanode, Minimal footprint in datanode code. Possible High level Approach: The ChunkManager and ChunkUtils can enable writeChunk for zero-filled chunks to be dropped without actually writing to the local filesystem. Similarly, if readChunk can construct a zero-filled buffer without reading from the local filesystem whenever it detects a zero-filled chunk. Specifics of how to detect and record a zero-filled chunk can be discussed on this jira. Also discuss how to control this behaviour and make it available only for internal testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1003) Intermittent IO exceptions encountered during pre-commit tests
[ https://issues.apache.org/jira/browse/HDDS-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh resolved HDDS-1003. - Resolution: Duplicate > Intermittent IO exceptions encountered during pre-commit tests > -- > > Key: HDDS-1003 > URL: https://issues.apache.org/jira/browse/HDDS-1003 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Supratim Deka >Priority: Major > > stack trace from > https://builds.apache.org/job/PreCommit-HDDS-Build/2095/testReport/org.apache.hadoop.ozone.client.rpc/TestOzoneRpcClient/testPutKey/ > java.io.IOException: Unexpected Storage Container Exception: > java.util.concurrent.ExecutionException: > org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io > exception > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunkToContainer(BlockOutputStream.java:622) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunk(BlockOutputStream.java:464) > at > org.apache.hadoop.hdds.scm.storage.BlockOutputStream.close(BlockOutputStream.java:480) > at > org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.close(BlockOutputStreamEntry.java:137) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleFlushOrClose(KeyOutputStream.java:481) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:314) > at > org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:258) > at > org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49) > at java.io.OutputStream.write(OutputStream.java:75) > at > org.apache.hadoop.ozone.client.rpc.TestOzoneRpcClientAbstract.testPutKey(TestOzoneRpcClientAbstract.java:522) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) > at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/ [Feb 11, 2019 3:17:08 AM] (bharat) HDDS-1078. TestRatisPipelineProvider failing because of node count [Feb 11, 2019 3:34:53 AM] (bharat) HDDS-1077. TestSecureOzoneCluster does not config OM HTTP keytab. [Feb 11, 2019 6:42:21 AM] (wwei) YARN-9252. Allocation Tag Namespace support in Distributed Shell. [Feb 11, 2019 7:53:50 AM] (wwei) YARN-8555. Parameterize TestSchedulingRequestContainerAllocation(Async) [Feb 11, 2019 11:18:26 AM] (elek) HDDS-1017. Use distributed tracing to indentify performance problems in [Feb 11, 2019 2:38:25 PM] (msingh) HDDS-1040. Add blockade Tests for client failures. Contributed by [Feb 11, 2019 6:09:44 PM] (inigoiri) HDFS-14260. Replace synchronized method in BlockReceiver with atomic [Feb 11, 2019 7:02:32 PM] (weichiu) HDFS-14261. Kerberize JournalNodeSyncer unit test. Contributed by Siyao [Feb 11, 2019 8:09:14 PM] (xyao) HDDS-1075. Fix CertificateUtil#parseRSAPublicKey charsetName. [Feb 11, 2019 9:07:47 PM] (iwasakims) HADOOP-16098. Fix javadoc warnings in hadoop-aws. Contributed by [Feb 11, 2019 10:54:30 PM] (sumasai) YARN-9229. Document docker registry deployment with NFS Gateway. [Feb 11, 2019 11:27:43 PM] (gifuma) YARN-996. REST API support for node resource configuration. Contributed -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.util.TestDiskChecker hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier hadoop.hdfs.TestFileCorruption hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.qjournal.server.TestJournalNodeSync hadoop.yarn.server.resourcemanager.scheduler.constraint.TestPlacementConstraintsUtil hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler hadoop.mapred.TestJobCounters hadoop.yarn.sls.TestSLSStreamAMSynth hadoop.yarn.sls.appmaster.TestAMSimulator cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/diff-compile-javac-root.txt [336K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/diff-checkstyle-root.txt [17M] hadolint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/diff-patch-hadolint.txt [8.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/diff-patch-pylint.txt [144K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/whitespace-eol.txt [9.6M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/whitespace-tabs.txt [1.1M] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/branch-findbugs-hadoop-hdds_client.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/branch-findbugs-hadoop-hdds_framework.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/branch-findbugs-hadoop-ozone_client.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/branch-findbugs-hadoop-ozone_common.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1045/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt [4.0K]
[jira] [Created] (HDDS-1093) Configuration tab in OM ui is not displaying the correct values
Sandeep Nemuri created HDDS-1093:
Summary: Configuration tab in OM ui is not displaying the correct
values
Key: HDDS-1093
URL: https://issues.apache.org/jira/browse/HDDS-1093
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: OM
Reporter: Sandeep Nemuri
Attachments: image-2019-02-12-19-47-18-332.png
Configuration tab in OM ui is not displaying the correct/configured values,
rather it is displaying the default values.
!image-2019-02-12-19-47-18-332.png!
{code:java}
[hdfs@freonnode10 hadoop]$ curl -s http://freonnode10:9874/conf | grep
ozone.om.handler.count.key
ozone.om.handler.count.key40falseozone-site.xml
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1092) Use Java 11 JRE to run Ozone in containers
Elek, Marton created HDDS-1092: -- Summary: Use Java 11 JRE to run Ozone in containers Key: HDDS-1092 URL: https://issues.apache.org/jira/browse/HDDS-1092 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Elek, Marton Assignee: Elek, Marton As of now we use opendk 1.8.0 in the Ozone containers. Java 9 and Java 10 introduces advanced support for the resource management of the containers and not all of them are available from the latest release of 1.8.0. (see this blog for more details: https://medium.com/adorsys/jvm-memory-settings-in-a-container-environment-64b0840e1d9e) I propose to switch to use Java 11 in the containers and test everything with Java 11 at runtime. Note: this issue is just about the runtime jdk not about the compile time JDK. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1091) Provide centos based apache/hadoop-runner base image for ozone.
Elek, Marton created HDDS-1091: -- Summary: Provide centos based apache/hadoop-runner base image for ozone. Key: HDDS-1091 URL: https://issues.apache.org/jira/browse/HDDS-1091 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Elek, Marton Assignee: Elek, Marton Our [current base image|https://github.com/apache/hadoop/blob/docker-hadoop-runner/Dockerfile] [is based on debian |https://github.com/apache/hadoop/blob/docker-hadoop-runner/Dockerfile](openjdk:8-jdk). Based on informal statistics the majority of the hadoop users and the potential Ozone users use centos/redhat. I propose to test Ozone with centos base image instead of debian one. There are two options here: 1. Provide two base images (one for debian one for centos) 2. Update the apache/hadoop-runner to use centos instead of debian (1) is more generic but it requires more work to maintain. (2) is more simple, but we need to test docker-hadoop-2 and docker-hadoop-3 branches to be sure that they work together with centos base image. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/ No changes -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-common-project/hadoop-common Class org.apache.hadoop.fs.GlobalStorageStatistics defines non-transient non-serializable instance field map In GlobalStorageStatistics.java:instance field map In GlobalStorageStatistics.java FindBugs : module:hadoop-hdfs-project/hadoop-hdfs Dead store to state in org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver.save(OutputStream, INodeSymlink) At FSImageFormatPBINode.java:org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver.save(OutputStream, INodeSymlink) At FSImageFormatPBINode.java:[line 623] FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:[line 335] Failed junit tests : hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.registry.secure.TestSecureLogins hadoop.yarn.server.nodemanager.containermanager.TestContainerManager hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 hadoop.yarn.client.api.impl.TestAMRMProxy cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt [328K] cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/diff-compile-cc-root-jdk1.8.0_191.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/diff-compile-javac-root-jdk1.8.0_191.txt [308K] checkstyle: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/diff-patch-shellcheck.txt [72K] shelldocs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/diff-patch-shelldocs.txt [8.0K] whitespace: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/whitespace-tabs.txt [1.2M] xml: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/xml.txt [8.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html [8.0K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html [8.0K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/230/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt [16K]
[jira] [Created] (HDDS-1090) Add a recipe to the ozone docs about using Spark, OzoneFS, and K8s
Elek, Marton created HDDS-1090: -- Summary: Add a recipe to the ozone docs about using Spark, OzoneFS, and K8s Key: HDDS-1090 URL: https://issues.apache.org/jira/browse/HDDS-1090 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Elek, Marton Assignee: Elek, Marton With the isolated classloader (HDDS-922) it's possible to use the regular spark distribution (which includes hadoop-2.7) and ozonefs (based on hadoop-3.2) together. In HDDS-846 we started to add '_recipes_' to the documentation to provide how-to style pages for different use-cases. I would add a new page about using the following components together: * Spark2.4-hadoop-2.7 * Ozonefs (isolated classloader) * Kubernetes spark executor -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1089) Disable OzoneFSStorageStatistics for hadoop versions older than 2.8
Elek, Marton created HDDS-1089: -- Summary: Disable OzoneFSStorageStatistics for hadoop versions older than 2.8 Key: HDDS-1089 URL: https://issues.apache.org/jira/browse/HDDS-1089 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: Ozone Filesystem Reporter: Elek, Marton Assignee: Elek, Marton HDDS-1033 introduced OzoneFSStorageStatistics for OzoneFileSystem. It uses the StorageStatistics which is introduced in HADOOP-13065 (available from the hadoop2.8/3.0). Using older hadoop (for example hadoop-2.7 which is included in the spark distributions) is not possible any more even with using the isolated class loader (introduced in HDDS-922). Fortunately it can be fixed: # We can support null in storageStatistics field with checking everywhere before call it. # We can create a new constructor of OzoneClientAdapterImpl without using OzoneFSStorageStatistics): If OzoneFSStorageStatistics is not in the method/constructor signature we don't need to load it. # We can check the availability of HADOOP-13065 and if the classes are not in the classpath we can skip the initialization of the OzoneFSStorageStatistics -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
