[jira] [Created] (HDFS-13821) RBF:add dfs.federation.router.mount-table.cache.enable so that users can disable cache
Fei Hui created HDFS-13821:
--
Summary: RBF:add dfs.federation.router.mount-table.cache.enable so
that users can disable cache
Key: HDFS-13821
URL: https://issues.apache.org/jira/browse/HDFS-13821
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs
Affects Versions: 3.0.3, 2.9.1, 3.1.0
Reporter: Fei Hui
Attachments: image-2018-08-13-11-27-49-023.png
When i test rbf, if found performance problem.
I found that ProxyAvgTime From Ganglia is so high, i run jstack on Router and
get the following stack frames
{quota}
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0005c264acd8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2249)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
at
org.apache.hadoop.hdfs.server.federation.resolver.MountTableResolver.getDestinationForPath(MountTableResolver.java:380)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2104)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:2087)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getListing(RouterRpcServer.java:1050)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:640)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
{/quota}
Many threads blocked on *LocalCache*
After disable the cache, ProxyAvgTime is down as follow showed
!image-2018-08-13-11-27-49-023.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13820) Disable CacheReplicationMonitor If No Cached Paths Exist
BELUGA BEHR created HDFS-13820:
--
Summary: Disable CacheReplicationMonitor If No Cached Paths Exist
Key: HDFS-13820
URL: https://issues.apache.org/jira/browse/HDFS-13820
Project: Hadoop HDFS
Issue Type: Improvement
Components: caching
Affects Versions: 2.10.0, 3.2.0
Reporter: BELUGA BEHR
Stating with [HDFS-6106] the loop for checking caching is set to be every 30
seconds.
Please implement a way to disable the {{CacheReplicationMonitor}} class if
there are no paths specified. Adding the first cached path to the NameNode
should kick off the {{CacheReplicationMonitor}} and when the last one is
deleted, the {{CacheReplicationMonitor}} should be disabled again.
Alternatively, provide a configuration flag to turn this feature off altogether.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/ No changes -1 overall The following subsystems voted -1: asflicense findbugs pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed CTEST tests : test_test_libhdfs_threaded_hdfs_static test_libhdfs_threaded_hdfspp_test_shim_static Failed junit tests : hadoop.hdfs.client.impl.TestBlockReaderLocal hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestMRTimelineEventHandling cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/diff-compile-javac-root.txt [328K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/diff-checkstyle-root.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/whitespace-eol.txt [9.4M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/whitespace-tabs.txt [1.1M] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/branch-findbugs-hadoop-hdds_client.txt [68K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/branch-findbugs-hadoop-hdds_framework.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [60K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/branch-findbugs-hadoop-ozone_client.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/branch-findbugs-hadoop-ozone_common.txt [28K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/branch-findbugs-hadoop-ozone_tools.txt [4.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/diff-javadoc-javadoc-root.txt [760K] CTEST: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt [116K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [432K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt [112K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [88K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/866/artifact/out/patch-unit-hadoop-hdds_client.txt [48K]
Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64
For more details, see https://builds.apache.org/job/hadoop-trunk-win/556/ [Aug 12, 2018 10:05:52 AM] (umamahesh) HDFS-10794. [SPS]: Provide storage policy satisfy worker at DN for [Aug 12, 2018 10:05:53 AM] (umamahesh) HDFS-10800: [SPS]: Daemon thread in Namenode to find blocks placed in [Aug 12, 2018 10:05:53 AM] (umamahesh) HDFS-10801. [SPS]: Protocol buffer changes for sending storage movement [Aug 12, 2018 10:05:53 AM] (umamahesh) HDFS-10884: [SPS]: Add block movement tracker to track the completion of [Aug 12, 2018 10:05:53 AM] (umamahesh) HDFS-10954. [SPS]: Provide mechanism to send blocks movement result back [Aug 12, 2018 10:05:53 AM] (umamahesh) HDFS-11029. [SPS]:Provide retry mechanism for the blocks which were [Aug 12, 2018 10:05:54 AM] (umamahesh) HDFS-11068: [SPS]: Provide unique trackID to track the block movement [Aug 12, 2018 10:05:54 AM] (umamahesh) HDFS-10802. [SPS]: Add satisfyStoragePolicy API in HdfsAdmin. [Aug 12, 2018 10:05:54 AM] (umamahesh) HDFS-11151. [SPS]: StoragePolicySatisfier should gracefully handle when [Aug 12, 2018 10:05:55 AM] (umamahesh) HDFS-10885. [SPS]: Mover tool should not be allowed to run when Storage [Aug 12, 2018 10:05:55 AM] (umamahesh) HDFS-11123. [SPS] Make storage policy satisfier daemon work on/off [Aug 12, 2018 10:05:55 AM] (umamahesh) HDFS-11032: [SPS]: Handling of block movement failure at the coordinator [Aug 12, 2018 10:05:55 AM] (umamahesh) HDFS-11248: [SPS]: Handle partial block location movements. Contributed [Aug 12, 2018 10:05:56 AM] (umamahesh) HDFS-11193 : [SPS]: Erasure coded files should be considered for [Aug 12, 2018 10:05:56 AM] (umamahesh) HDFS-11289. [SPS]: Make SPS movement monitor timeouts configurable. [Aug 12, 2018 10:05:56 AM] (umamahesh) HDFS-11293: [SPS]: Local DN should be given preference as source node, [Aug 12, 2018 10:05:57 AM] (umamahesh) HDFS-11150: [SPS]: Provide persistence when satisfying storage policy. [Aug 12, 2018 10:05:57 AM] (umamahesh) HDFS-11186. [SPS]: Daemon thread of SPS should start only in Active NN. [Aug 12, 2018 10:05:57 AM] (umamahesh) HDFS-11309. [SPS]: chooseTargetTypeInSameNode should pass accurate block [Aug 12, 2018 10:05:57 AM] (umamahesh) HDFS-11243. [SPS]: Add a protocol command from NN to DN for dropping the [Aug 12, 2018 10:05:57 AM] (umamahesh) HDFS-11239: [SPS]: Check Mover file ID lease also to determine whether [Aug 12, 2018 10:05:58 AM] (umamahesh) HDFS-11336: [SPS]: Remove xAttrs when movements done or SPS disabled. [Aug 12, 2018 10:05:58 AM] (umamahesh) HDFS-11338: [SPS]: Fix timeout issue in unit tests caused by longger NN [Aug 12, 2018 10:05:58 AM] (umamahesh) HDFS-11334: [SPS]: NN switch and rescheduling movements can lead to have [Aug 12, 2018 10:05:58 AM] (umamahesh) HDFS-11572. [SPS]: SPS should clean Xattrs when no blocks required to [Aug 12, 2018 10:05:59 AM] (umamahesh) HDFS-11695: [SPS]: Namenode failed to start while loading SPS xAttrs [Aug 12, 2018 10:05:59 AM] (umamahesh) HDFS-11883: [SPS] : Handle NPE in BlockStorageMovementTracker when [Aug 12, 2018 10:05:59 AM] (umamahesh) HDFS-11762. [SPS]: Empty files should be ignored in [Aug 12, 2018 10:05:59 AM] (umamahesh) HDFS-11726. [SPS]: StoragePolicySatisfier should not select same storage [Aug 12, 2018 10:05:59 AM] (umamahesh) HDFS-11966. [SPS] Correct the log in [Aug 12, 2018 10:05:59 AM] (umamahesh) HDFS-11670: [SPS]: Add CLI command for satisfy storage policy [Aug 12, 2018 10:06:00 AM] (umamahesh) HDFS-11965: [SPS]: Should give chance to satisfy the low redundant [Aug 12, 2018 10:06:00 AM] (umamahesh) HDFS-11264: [SPS]: Double checks to ensure that SPS/Mover are not [Aug 12, 2018 10:06:00 AM] (umamahesh) HDFS-11874. [SPS]: Document the SPS feature. Contributed by Uma [Aug 12, 2018 10:06:00 AM] (umamahesh) HDFS-12146. [SPS]: Fix [Aug 12, 2018 10:06:00 AM] (umamahesh) HDFS-12141: [SPS]: Fix checkstyle warnings. Contributed by Rakesh R. [Aug 12, 2018 10:06:00 AM] (umamahesh) HDFS-12152: [SPS]: Re-arrange StoragePolicySatisfyWorker stopping [Aug 12, 2018 10:06:01 AM] (umamahesh) HDFS-12214: [SPS]: Fix review comments of StoragePolicySatisfier [Aug 12, 2018 10:06:01 AM] (umamahesh) HDFS-12225: [SPS]: Optimize extended attributes for tracking SPS [Aug 12, 2018 10:06:01 AM] (umamahesh) HDFS-12291: [SPS]: Provide a mechanism to recursively iterate and [Aug 12, 2018 10:06:01 AM] (umamahesh) HDFS-12570: [SPS]: Refactor Co-ordinator datanode logic to track the [Aug 12, 2018 10:06:01 AM] (umamahesh) HDFS-12556: [SPS] : Block movement analysis should be done in read lock. [Aug 12, 2018 10:06:02 AM] (umamahesh) HDFS-12310: [SPS]: Provide an option to track the status of in progress [Aug 12, 2018 10:06:02 AM] (umamahesh) HDFS-12790: [SPS]: Rebasing HDFS-10285 branch after HDFS-10467, [Aug 12, 2018 10:06:02 AM] (umamahesh) HDFS-12106: [SPS]: Improve storage policy satisfier configurations. [Aug 12, 2018 10:06:02 AM] (umamahesh) HDFS-12955: [SPS]: Move SPS classes to a separate package.
[jira] [Created] (HDDS-348) Parallalize container ops in ContainerStateMachine
Mukul Kumar Singh created HDDS-348: -- Summary: Parallalize container ops in ContainerStateMachine Key: HDDS-348 URL: https://issues.apache.org/jira/browse/HDDS-348 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode Affects Versions: 0.2.1 Reporter: Mukul Kumar Singh Assignee: Mukul Kumar Singh Fix For: 0.2.1 Currently all the ops in ContainerStateMachine#applyTransaction are processed sequentially by the ContainerStateMachine. However these ops can be parallelized by having a per container executor. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
