[jira] [Commented] (HDFS-13044) RBF: Add a safe mode for the Router
[
https://issues.apache.org/jira/browse/HDFS-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340632#comment-16340632
]
Yiqun Lin commented on HDFS-13044:
--
Almost looks good to me and your comment makes sense to me. Only some minor
comments relevant to the safemode settings:
*DFSConfigKeys.java*
line1240. Would you set {{DFS_ROUTER_CACHE_TIME_TO_LIVE_MS_DEFAULT * 3}} as
the default value? Seems this haven't been addressed.
*RouterSafemodeService.java*
line84 and line132: Would you print the time with {{MILLISECONDS}} format? The
potential problem of accurate loosing.
line101: Can you update the description of setting
{{dfs.federation.router.cache.ttl}} and change default value to {{1m}}? As I
see you have made it supporting time unit suffixes.
Others looks good to me.
> RBF: Add a safe mode for the Router
> ---
>
> Key: HDFS-13044
> URL: https://issues.apache.org/jira/browse/HDFS-13044
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-13004.000.patch, HDFS-13044.001.patch,
> HDFS-13044.002.patch, HDFS-13044.003.patch
>
>
> When a Router cannot communicate with the State Store, it should enter into a
> safe mode that disallows certain operations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13057) [SPS]: Revisit configurations to make SPS service modes internal/external/none
[
https://issues.apache.org/jira/browse/HDFS-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340628#comment-16340628
]
Rakesh R commented on HDFS-13057:
-
Thanks [~umamaheswararao] for the reviews. Attached another patch addressing
the above comments.
> [SPS]: Revisit configurations to make SPS service modes internal/external/none
> --
>
> Key: HDFS-13057
> URL: https://issues.apache.org/jira/browse/HDFS-13057
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Blocker
> Attachments: HDFS-13057-HDFS-10285-00.patch,
> HDFS-13057-HDFS-10285-01.patch, HDFS-13057-HDFS-10285-02.patch
>
>
> This task is to revisit the configurations to make SPS service modes -
> {{internal/external/none}}
> - {{internal}} : represents SPS service will be running with NN
> - {{external}}: represents SPS service will be running outside NN
> - {{none}}: represents the SPS service is completely disabled and zero cost
> to the system.
> Proposed configuration {{dfs.storage.policy.satisfier.mode}} item in
> hdfs-site.xml file and value will be string. The mode can be changed via
> {{reconfig}} command.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13057) [SPS]: Revisit configurations to make SPS service modes internal/external/none
[
https://issues.apache.org/jira/browse/HDFS-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-13057:
Attachment: HDFS-13057-HDFS-10285-02.patch
> [SPS]: Revisit configurations to make SPS service modes internal/external/none
> --
>
> Key: HDFS-13057
> URL: https://issues.apache.org/jira/browse/HDFS-13057
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Blocker
> Attachments: HDFS-13057-HDFS-10285-00.patch,
> HDFS-13057-HDFS-10285-01.patch, HDFS-13057-HDFS-10285-02.patch
>
>
> This task is to revisit the configurations to make SPS service modes -
> {{internal/external/none}}
> - {{internal}} : represents SPS service will be running with NN
> - {{external}}: represents SPS service will be running outside NN
> - {{none}}: represents the SPS service is completely disabled and zero cost
> to the system.
> Proposed configuration {{dfs.storage.policy.satisfier.mode}} item in
> hdfs-site.xml file and value will be string. The mode can be changed via
> {{reconfig}} command.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13021) Incorrect storage policy of snapshot file was returned by getStoragePolicy command
[
https://issues.apache.org/jira/browse/HDFS-13021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340602#comment-16340602
]
Xiao Chen commented on HDFS-13021:
--
Thanks [~GeLiXin] for reporting the issue and providing a patch. Looks like
there are some unnecessary space-only changes (e.g. FSDirStatAndListingOp),
could you get rid of them in the next rev? No need to roll a version only for
this though.
Thinking about this more, this appear to be trickier than EC / EZ since the
blocks may be moved, but OTOH snapshots are supposed to be immutable. And if a
block is only in snapshot then it will not be moved because the mover can't see
it from the file name.
Hi [~szetszwo],
What's your take on this one? Is this behavior by design or should be improved
in some way?
> Incorrect storage policy of snapshot file was returned by getStoragePolicy
> command
> --
>
> Key: HDFS-13021
> URL: https://issues.apache.org/jira/browse/HDFS-13021
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, snapshots
>Affects Versions: 3.1.0
>Reporter: LiXin Ge
>Assignee: LiXin Ge
>Priority: Major
> Attachments: HDFS-13021.001.patch
>
>
> Snapshots are supposed to be immutable and read only, so the file status
> which in a snapshot path shouldn't follow the original file's change.
> The StoragePolicy in snapshot situation acts like a bug now.
> ---
> Reproduction：Operation on snapshottable dir {{/storagePolicy}}
> *before make snapshot:*
> {code:java}
> [bin]# hdfs storagepolicies -setStoragePolicy -path /storagePolicy -policy
> PROVIDED
> Set storage policy PROVIDED on /storagePolicy
> [bin]# hadoop fs -put /home/file /storagePolicy/file_PROVIDED
> [bin]# hdfs storagepolicies -getStoragePolicy -path
> /storagePolicy/file_PROVIDED
> The storage policy of /storagePolicy/file_PROVIDED:
> BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[],
> replicationFallbacks=[ARCHIVE]}
> {code}
> *make snapshot and check:*
> {code:java}
> [bin]# hdfs dfs -createSnapshot /storagePolicy s3_PROVIDED
> Created snapshot /storagePolicy/.snapshot/s3_PROVIDED
> [bin]# hdfs storagepolicies -getStoragePolicy -path
> /storagePolicy/.snapshot/s3_PROVIDED/file_PROVIDED
> The storage policy of /storagePolicy/.snapshot/s3_PROVIDED/file_PROVIDED:
> BlockStoragePolicy{PROVIDED:1, storageTypes=[PROVIDED, DISK],
> creationFallbacks=[PROVIDED, DISK], replicationFallbacks=[PROVIDED, DISK]}
> {code}
> *change the StroagePolicy and check again:*
> {code:java}
> [bin]# hdfs storagepolicies -setStoragePolicy -path /storagePolicy -policy HOT
> Set storage policy HOT on /storagePolicy
> [bin]# hdfs storagepolicies -getStoragePolicy -path
> /storagePolicy/.snapshot/s3_PROVIDED/file_PROVIDED
> The storage policy of /storagePolicy/.snapshot/s3_PROVIDED/file_PROVIDED:
> BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[],
> replicationFallbacks=[ARCHIVE]} It shouldn't be HOT
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13068) RBF: Add router admin option to manage safe mode
[ https://issues.apache.org/jira/browse/HDFS-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin reassigned HDFS-13068: Assignee: Yiqun Lin > RBF: Add router admin option to manage safe mode > > > Key: HDFS-13068 > URL: https://issues.apache.org/jira/browse/HDFS-13068 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Yiqun Lin >Priority: Minor > > HDFS-13044 adds a safe mode to reject requests. We should have an option to > manually set the Router into safe mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13068) RBF: Add router admin option to manage safe mode
[ https://issues.apache.org/jira/browse/HDFS-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340600#comment-16340600 ] Yiqun Lin commented on HDFS-13068: -- Yeah, assign to myself, :). > RBF: Add router admin option to manage safe mode > > > Key: HDFS-13068 > URL: https://issues.apache.org/jira/browse/HDFS-13068 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Priority: Minor > > HDFS-13044 adds a safe mode to reject requests. We should have an option to > manually set the Router into safe mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13065) TestErasureCodingMultipleRacks#testSkewedRack3 is failing
[ https://issues.apache.org/jira/browse/HDFS-13065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340599#comment-16340599 ] Xiao Chen commented on HDFS-13065: -- Thanks [~gabor.bota] for reporting and fixing the test here! The indent-only change at line 182 seems unnecessary. +1 pending that and the last checkstyle. > TestErasureCodingMultipleRacks#testSkewedRack3 is failing > - > > Key: HDFS-13065 > URL: https://issues.apache.org/jira/browse/HDFS-13065 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Fix For: 3.0.1 > > Attachments: HDFS-13065.001.patch, HDFS-13065.002.patch > > > Test fails intermittently. > java.lang.AssertionError: expected:<9> but was:<7> > at > org.apache.hadoop.hdfs.TestErasureCodingMultipleRacks.testSkewedRack3(TestErasureCodingMultipleRacks.java:178) > 2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] DEBUG > net.NetworkTopology > (DFSNetworkTopology.java:chooseRandomWithStorageTypeTwoTrial(119)) - No node > to choose. > 2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] WARN > blockmanagement.BlockPlacementPolicy > (BlockPlacementPolicyRackFaultTolerant.java:chooseTargetInOrder(142)) - Only > able to place 7 of total expected 9 (maxNodesPerRack=2, numOfReplicas=4) > nodes evenly across racks, falling back to evenly place on the remaining > racks. This may not guarantee rack-level fault tolerance. Please check if the > racks are configured properly. > 2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] DEBUG > blockmanagement.BlockPlacementPolicy > (BlockPlacementPolicyRackFaultTolerant.java:chooseTargetInOrder(148)) - > Caught exception was: > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException: > > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:792) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant.chooseOnce(BlockPlacementPolicyRackFaultTolerant.java:224) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant.chooseTargetInOrder(BlockPlacementPolicyRackFaultTolerant.java:139) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:392) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:268) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:121) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:137) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2091) > at > org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:287) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2602) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:864) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:549) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12974) Exception information can not be returned when I create transparent encryption zone.
[ https://issues.apache.org/jira/browse/HDFS-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340597#comment-16340597 ] fang zhenyi commented on HDFS-12974: Thanks [~xiaochen] for reviewing my patch,sorry I forgot to add. I have added in HDFS-12974.009.patch.Hope you can review again, thanks a lot. > Exception information can not be returned when I create transparent > encryption zone. > > > Key: HDFS-12974 > URL: https://issues.apache.org/jira/browse/HDFS-12974 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 3.0.0 >Reporter: fang zhenyi >Assignee: fang zhenyi >Priority: Minor > Attachments: HDFS-12974.001.patch, HDFS-12974.002.patch, > HDFS-12974.003.patch, HDFS-12974.004.patch, HDFS-12974.005.patch, > HDFS-12974.006.patch, HDFS-12974.007.patch, HDFS-12974.008.patch, > HDFS-12974.009.patch > > > When I add the following configuration to the kms-acl.xml file, I create > encrypted space and I can not get any exception information. > > key.acl.key2.GENERATE_EEK > mr > > root@fangzhenyi01:~# hdfs crypto -createZone -keyName key2 -path /zone > 2018-01-02 10:41:44,632 WARN util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > RemoteException: > root@fangzhenyi01:~# -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12897) Path not found when we get the ec policy for a .snapshot dir
[
https://issues.apache.org/jira/browse/HDFS-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340596#comment-16340596
]
Xiao Chen commented on HDFS-12897:
--
Thanks [~GeLiXin] for the new rev, and sorry for my delayed review here.
Patch 4 LGTM overall. +1 pending 2 nits:
- TestErasureCodingPolicyWithSnapshot: {{final Path snap1 =
fs.createSnapshot(ecDir, "snap0");}} change snap0 to snap1 to be consistent
with the rest of the test
- TestErasureCodingPolicyWithSnapshot: we can set a test class level {{Rule}},
that way we don't have to set timeout on each test cases.
> Path not found when we get the ec policy for a .snapshot dir
>
>
> Key: HDFS-12897
> URL: https://issues.apache.org/jira/browse/HDFS-12897
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs, snapshots
>Affects Versions: 3.0.0-alpha1, 3.1.0
>Reporter: Harshakiran Reddy
>Assignee: LiXin Ge
>Priority: Major
> Attachments: HDFS-12897.001.patch, HDFS-12897.002.patch,
> HDFS-12897.003.patch, HDFS-12897.004.patch
>
>
> Scenario:-
> ---
> Operation on snapshot dir.
> *EC policy*
> bin> ./hdfs ec -getPolicy -path /dir/
> RS-3-2-1024k
> bin> ./hdfs ec -getPolicy -path /dir/.snapshot/
> {{FileNotFoundException: Path not found: /dir/.snapshot}}
> bin> ./hdfs dfs -ls /dir/.snapshot/
> Found 2 items
> drwxr-xr-x - user group 0 2017-12-05 12:27 /dir/.snapshot/s1
> drwxr-xr-x - user group 0 2017-12-05 12:28 /dir/.snapshot/s2
> *Storagepolicies*
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/.snapshot/
> {{The storage policy of /dir/.snapshot/ is unspecified}}
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/
> The storage policy of /dir/:
> BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[],
> replicationFallbacks=[]}
> *Which is the correct behavior ?*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12997) Move logging to slf4j in BlockPoolSliceStorage and Storage
[
https://issues.apache.org/jira/browse/HDFS-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340594#comment-16340594
]
Akira Ajisaka commented on HDFS-12997:
--
Thanks! I'm +1 if the following comments are addressed.
1. {{#toString}} method call should be removed from the arguments of logging
functions to avoid redundant toString method calls by lazy evaluation. For
example, in Storage.java line 901:
{code:java}LOG.error("Unable to acquire file lock on path {}",
lockF.toString());{code}
{{lockF.toString()}} can be replaced with {{LockF}}. Similar issues exist in
DataStorage.java line 672 and NNStorage.java line 1156.
2. DataStorage.java line 1238: Missing period in the end of warn message.
> Move logging to slf4j in BlockPoolSliceStorage and Storage
> ---
>
> Key: HDFS-12997
> URL: https://issues.apache.org/jira/browse/HDFS-12997
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-12997.001.patch, HDFS-12997.002.patch,
> HDFS-12997.003.patch, HDFS-12997.004.patch
>
>
> Move logging to slf4j in BlockPoolSliceStorage and Storage classes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12974) Exception information can not be returned when I create transparent encryption zone.
[ https://issues.apache.org/jira/browse/HDFS-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fang zhenyi updated HDFS-12974: --- Status: Patch Available (was: In Progress) > Exception information can not be returned when I create transparent > encryption zone. > > > Key: HDFS-12974 > URL: https://issues.apache.org/jira/browse/HDFS-12974 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 3.0.0 >Reporter: fang zhenyi >Assignee: fang zhenyi >Priority: Minor > Attachments: HDFS-12974.001.patch, HDFS-12974.002.patch, > HDFS-12974.003.patch, HDFS-12974.004.patch, HDFS-12974.005.patch, > HDFS-12974.006.patch, HDFS-12974.007.patch, HDFS-12974.008.patch, > HDFS-12974.009.patch > > > When I add the following configuration to the kms-acl.xml file, I create > encrypted space and I can not get any exception information. > > key.acl.key2.GENERATE_EEK > mr > > root@fangzhenyi01:~# hdfs crypto -createZone -keyName key2 -path /zone > 2018-01-02 10:41:44,632 WARN util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > RemoteException: > root@fangzhenyi01:~# -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12974) Exception information can not be returned when I create transparent encryption zone.
[ https://issues.apache.org/jira/browse/HDFS-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fang zhenyi updated HDFS-12974: --- Attachment: HDFS-12974.009.patch > Exception information can not be returned when I create transparent > encryption zone. > > > Key: HDFS-12974 > URL: https://issues.apache.org/jira/browse/HDFS-12974 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 3.0.0 >Reporter: fang zhenyi >Assignee: fang zhenyi >Priority: Minor > Attachments: HDFS-12974.001.patch, HDFS-12974.002.patch, > HDFS-12974.003.patch, HDFS-12974.004.patch, HDFS-12974.005.patch, > HDFS-12974.006.patch, HDFS-12974.007.patch, HDFS-12974.008.patch, > HDFS-12974.009.patch > > > When I add the following configuration to the kms-acl.xml file, I create > encrypted space and I can not get any exception information. > > key.acl.key2.GENERATE_EEK > mr > > root@fangzhenyi01:~# hdfs crypto -createZone -keyName key2 -path /zone > 2018-01-02 10:41:44,632 WARN util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > RemoteException: > root@fangzhenyi01:~# -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12974) Exception information can not be returned when I create transparent encryption zone.
[ https://issues.apache.org/jira/browse/HDFS-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fang zhenyi updated HDFS-12974: --- Status: In Progress (was: Patch Available) > Exception information can not be returned when I create transparent > encryption zone. > > > Key: HDFS-12974 > URL: https://issues.apache.org/jira/browse/HDFS-12974 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 3.0.0 >Reporter: fang zhenyi >Assignee: fang zhenyi >Priority: Minor > Attachments: HDFS-12974.001.patch, HDFS-12974.002.patch, > HDFS-12974.003.patch, HDFS-12974.004.patch, HDFS-12974.005.patch, > HDFS-12974.006.patch, HDFS-12974.007.patch, HDFS-12974.008.patch > > > When I add the following configuration to the kms-acl.xml file, I create > encrypted space and I can not get any exception information. > > key.acl.key2.GENERATE_EEK > mr > > root@fangzhenyi01:~# hdfs crypto -createZone -keyName key2 -path /zone > 2018-01-02 10:41:44,632 WARN util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > RemoteException: > root@fangzhenyi01:~# -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12997) Move logging to slf4j in BlockPoolSliceStorage and Storage
[
https://issues.apache.org/jira/browse/HDFS-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340582#comment-16340582
]
genericqa commented on HDFS-12997:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs generated 0 new +
392 unchanged - 2 fixed = 392 total (was 394) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 152 unchanged - 7 fixed = 153 total (was 159) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}118m 22s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12997 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907806/HDFS-12997.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 46e4cd4f3f71 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2e58656 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/22826/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22826/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-12974) Exception information can not be returned when I create transparent encryption zone.
[
https://issues.apache.org/jira/browse/HDFS-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340569#comment-16340569
]
Xiao Chen commented on HDFS-12974:
--
Thanks for revving [~zhenyi].
Any reason you skipped Rushabh's original comment about below?
{code:java}
public void printStackTrace() {
printStackTrace(System.err);
}{code}
I think this will be more consistent across the 3 methods.
+1 once this is addressed
> Exception information can not be returned when I create transparent
> encryption zone.
>
>
> Key: HDFS-12974
> URL: https://issues.apache.org/jira/browse/HDFS-12974
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 3.0.0
>Reporter: fang zhenyi
>Assignee: fang zhenyi
>Priority: Minor
> Attachments: HDFS-12974.001.patch, HDFS-12974.002.patch,
> HDFS-12974.003.patch, HDFS-12974.004.patch, HDFS-12974.005.patch,
> HDFS-12974.006.patch, HDFS-12974.007.patch, HDFS-12974.008.patch
>
>
> When I add the following configuration to the kms-acl.xml file, I create
> encrypted space and I can not get any exception information.
>
> key.acl.key2.GENERATE_EEK
> mr
>
> root@fangzhenyi01:~# hdfs crypto -createZone -keyName key2 -path /zone
> 2018-01-02 10:41:44,632 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> RemoteException:
> root@fangzhenyi01:~#
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340558#comment-16340558
]
Xiao Chen commented on HDFS-12528:
--
Thanks for looking at this [~GeLiXin] and offering to work on it.
I only did #1 because that's easiest and the most compatible way. Similar to #2
being subjective, #3 also varies based on the exception. It may turn out in the
future that some other exceptions should not disable SCR too, which will
require more change. But if you want to give #3 a try, please feel free to, I
can help with reviews then. If that takes time we can also separate out a new
jira.
[~jzhuge] and [~cheersyang], any thoughts from you?
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HDFS-12528.000.patch, HDFS-12528.01.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12522) Ozone: Remove the Priority Queues used in the Container State Manager
[ https://issues.apache.org/jira/browse/HDFS-12522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340546#comment-16340546 ] Xiaoyu Yao commented on HDFS-12522: --- Thanks [~anu] for the update. Looks like a few tests are failing now with "Lease Exception." during container state update. Can you take a look? > Ozone: Remove the Priority Queues used in the Container State Manager > - > > Key: HDFS-12522 > URL: https://issues.apache.org/jira/browse/HDFS-12522 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > Attachments: HDFS-12522-HDFS-7240.001.patch, > HDFS-12522-HDFS-7240.002.patch, HDFS-12522-HDFS-7240.003.patch > > > During code review of HDFS-12387, it was suggested that we remove the > priority queues that was used in ContainerStateManager. This JIRA tracks that > issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13044) RBF: Add a safe mode for the Router
[
https://issues.apache.org/jira/browse/HDFS-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340509#comment-16340509
]
genericqa commented on HDFS-13044:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}125m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}172m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| | hadoop.hdfs.TestFileAppend3 |
| | hadoop.hdfs.TestAppendSnapshotTruncate |
| | hadoop.hdfs.TestDFSPermission |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 |
| | hadoop.hdfs.TestParallelShortCircuitReadUnCached |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestBlockStoragePolicy |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.qjournal.client.TestQJMWithFaults |
| | hadoop.hdfs.TestDFSStorageStateRecovery |
| |
[jira] [Commented] (HDFS-13043) RBF: Expose the state of the Routers in the federation
[
https://issues.apache.org/jira/browse/HDFS-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340486#comment-16340486
]
genericqa commented on HDFS-13043:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 17s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}140m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13043 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907791/HDFS-13043.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 819491b06d76 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ff8378e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22823/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22823/testReport/ |
| Max. process+thread count | 3955 (vs. ulimit of 5000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/22823/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
[jira] [Commented] (HDFS-12997) Move logging to slf4j in BlockPoolSliceStorage and Storage
[ https://issues.apache.org/jira/browse/HDFS-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340467#comment-16340467 ] Ajay Kumar commented on HDFS-12997: --- [~xyao], thanks for review. Updated patch to address the suggestions. > Move logging to slf4j in BlockPoolSliceStorage and Storage > --- > > Key: HDFS-12997 > URL: https://issues.apache.org/jira/browse/HDFS-12997 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-12997.001.patch, HDFS-12997.002.patch, > HDFS-12997.003.patch, HDFS-12997.004.patch > > > Move logging to slf4j in BlockPoolSliceStorage and Storage classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12997) Move logging to slf4j in BlockPoolSliceStorage and Storage
[ https://issues.apache.org/jira/browse/HDFS-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12997: -- Attachment: HDFS-12997.004.patch > Move logging to slf4j in BlockPoolSliceStorage and Storage > --- > > Key: HDFS-12997 > URL: https://issues.apache.org/jira/browse/HDFS-12997 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-12997.001.patch, HDFS-12997.002.patch, > HDFS-12997.003.patch, HDFS-12997.004.patch > > > Move logging to slf4j in BlockPoolSliceStorage and Storage classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13069) Enable HDFS to cache data read from external storage systems
Virajith Jalaparti created HDFS-13069:
-
Summary: Enable HDFS to cache data read from external storage
systems
Key: HDFS-13069
URL: https://issues.apache.org/jira/browse/HDFS-13069
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Virajith Jalaparti
With {{PROVIDED}} storage (HDFS-9806), HDFS can address data stored in external
storage systems. Caching this data, locally, in HDFS can speed up subsequent
accesses to the data -- this feature is especially useful when accesses to the
external store have limited bandwidth/higher latency. This JIRA is to add this
feature in HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12522) Ozone: Remove the Priority Queues used in the Container State Manager
[
https://issues.apache.org/jira/browse/HDFS-12522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340456#comment-16340456
]
genericqa commented on HDFS-12522:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
56s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 57s{color}
| {color:red} hadoop-hdfs-project generated 12 new + 434 unchanged - 0 fixed =
446 total (was 434) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 44s{color} | {color:orange} hadoop-hdfs-project: The patch generated 8 new +
9 unchanged - 0 fixed = 17 total (was 9) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
41s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}151m 22s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}240m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.scm.TestContainerSQLCli |
| | hadoop.ozone.web.client.TestKeysRatis |
| | hadoop.ozone.ozShell.TestOzoneShell |
| | hadoop.ozone.TestOzoneConfigurationFields |
| | hadoop.ozone.web.TestOzoneRestWithMiniCluster |
| | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.ozone.ksm.TestMultipleContainerReadWrite |
| |
[jira] [Commented] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340427#comment-16340427
]
genericqa commented on HDFS-12528:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-hdfs-project: The patch generated 7 new +
90 unchanged - 1 fixed = 97 total (was 91) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 46s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
27s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}113m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}192m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12528 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/1290/HDFS-12528.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 56262eec8b2e 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 16be42d |
| maven | version: Apache Maven 3.3.9 |
| Default
[jira] [Commented] (HDFS-13044) RBF: Add a safe mode for the Router
[
https://issues.apache.org/jira/browse/HDFS-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340425#comment-16340425
]
genericqa commented on HDFS-13044:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 431 unchanged - 0 fixed = 432 total (was 431) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}124m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}172m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13044 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907779/HDFS-13044.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 2f78b995df0c 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 16be42d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/22821/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
[jira] [Commented] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340400#comment-16340400
]
genericqa commented on HDFS-12051:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 33m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 1235 unchanged - 18 fixed = 1237 total (was 1253) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
10s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}175m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Increment of volatile field
org.apache.hadoop.hdfs.server.namenode.NameCache.size in
org.apache.hadoop.hdfs.server.namenode.NameCache.put(byte[]) At
NameCache.java:in org.apache.hadoop.hdfs.server.namenode.NameCache.put(byte[])
At NameCache.java:[line 116] |
| Failed junit tests | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12051 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907776/HDFS-12051.08.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 1a0589f35cc2 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340385#comment-16340385
]
LiXin Ge commented on HDFS-12528:
-
BTW, [~xiaochen] in case if you don't have enough time to do #3, I'm willing to
have a try on it.
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HDFS-12528.000.patch, HDFS-12528.01.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340358#comment-16340358
]
LiXin Ge commented on HDFS-12528:
-
IMO, how about to do both #1 and #3? #1 make sure that user can avoid
overreaction when FNFE occur, but FNFE is just one of the exceptions which may
happens in short-circuit reads,what if there were two type of exceptions
happens in the same cluster, one is acceptable FNFE and the other is
unacceptable that user should disable the short-circuit reads for a while?
Ignore the acceptable FNFE and set a appropriate disable time to handle the
unacceptable exceptions seems better in this situation.
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HDFS-12528.000.patch, HDFS-12528.01.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13043) RBF: Expose the state of the Routers in the federation
[ https://issues.apache.org/jira/browse/HDFS-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340346#comment-16340346 ] Íñigo Goiri commented on HDFS-13043: I think I fixed all the issues reported by Yetus in [^HDFS-13043.001.patch]. A screenshot also attached: !router-info.png! > RBF: Expose the state of the Routers in the federation > -- > > Key: HDFS-13043 > URL: https://issues.apache.org/jira/browse/HDFS-13043 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13043.000.patch, HDFS-13043.001.patch, > router-info.png > > > The Router should expose the state of the other Routers in the federation > through a user UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13043) RBF: Expose the state of the Routers in the federation
[ https://issues.apache.org/jira/browse/HDFS-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13043: --- Attachment: router-info.png > RBF: Expose the state of the Routers in the federation > -- > > Key: HDFS-13043 > URL: https://issues.apache.org/jira/browse/HDFS-13043 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13043.000.patch, HDFS-13043.001.patch, > router-info.png > > > The Router should expose the state of the other Routers in the federation > through a user UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13043) RBF: Expose the state of the Routers in the federation
[ https://issues.apache.org/jira/browse/HDFS-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13043: --- Attachment: HDFS-13043.001.patch > RBF: Expose the state of the Routers in the federation > -- > > Key: HDFS-13043 > URL: https://issues.apache.org/jira/browse/HDFS-13043 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13043.000.patch, HDFS-13043.001.patch > > > The Router should expose the state of the other Routers in the federation > through a user UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13044) RBF: Add a safe mode for the Router
[ https://issues.apache.org/jira/browse/HDFS-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13044: --- Attachment: HDFS-13044.003.patch > RBF: Add a safe mode for the Router > --- > > Key: HDFS-13044 > URL: https://issues.apache.org/jira/browse/HDFS-13044 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13004.000.patch, HDFS-13044.001.patch, > HDFS-13044.002.patch, HDFS-13044.003.patch > > > When a Router cannot communicate with the State Store, it should enter into a > safe mode that disallows certain operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13068) RBF: Add router admin option to manage safe mode
[ https://issues.apache.org/jira/browse/HDFS-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340332#comment-16340332 ] Íñigo Goiri commented on HDFS-13068: [~linyiqun], do you mind taking this one? > RBF: Add router admin option to manage safe mode > > > Key: HDFS-13068 > URL: https://issues.apache.org/jira/browse/HDFS-13068 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Priority: Minor > > HDFS-13044 adds a safe mode to reject requests. We should have an option to > manually set the Router into safe mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13044) RBF: Add a safe mode for the Router
[
https://issues.apache.org/jira/browse/HDFS-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340331#comment-16340331
]
genericqa commented on HDFS-13044:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
41s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
33s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
37s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m
45s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
37s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
3s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 2 new + 1
unchanged - 0 fixed = 3 total (was 1) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13044 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907782/HDFS-13044.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux bb46e670b5c2 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 16be42d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/22822/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/22822/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| javac |
[jira] [Created] (HDFS-13068) RBF: Add router admin option to manage safe mode
Íñigo Goiri created HDFS-13068: -- Summary: RBF: Add router admin option to manage safe mode Key: HDFS-13068 URL: https://issues.apache.org/jira/browse/HDFS-13068 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Íñigo Goiri -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13068) RBF: Add router admin option to manage safe mode
[ https://issues.apache.org/jira/browse/HDFS-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13068: --- Description: HDFS-13044 adds a safe mode to reject requests. We should have an option to manually set the Router into safe mode. > RBF: Add router admin option to manage safe mode > > > Key: HDFS-13068 > URL: https://issues.apache.org/jira/browse/HDFS-13068 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Priority: Minor > > HDFS-13044 adds a safe mode to reject requests. We should have an option to > manually set the Router into safe mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-12997) Move logging to slf4j in BlockPoolSliceStorage and Storage
[
https://issues.apache.org/jira/browse/HDFS-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru updated HDFS-12997:
--
Comment: was deleted
(was: Thanks [~ajayydv] for working on this.
# In {{DataStorage#doRollback(), }} there is a missing {}.
{code:java}
LOG.info("Rolling back storage directory .\n target LV = {}; target "
+ "CTime = {}", sd.getRoot(), nsInfo.getCTime(),
HdfsServerConstants.DATANODE_LAYOUT_VERSION);
{code}
# Nitpick: In {{DataStorage#removeDuplicateEntries()}}, line 1239, there is a
missing '.' (Just in case someone is running a log scanner with exact string
match :)).)
> Move logging to slf4j in BlockPoolSliceStorage and Storage
> ---
>
> Key: HDFS-12997
> URL: https://issues.apache.org/jira/browse/HDFS-12997
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-12997.001.patch, HDFS-12997.002.patch,
> HDFS-12997.003.patch
>
>
> Move logging to slf4j in BlockPoolSliceStorage and Storage classes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12997) Move logging to slf4j in BlockPoolSliceStorage and Storage
[
https://issues.apache.org/jira/browse/HDFS-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340307#comment-16340307
]
Hanisha Koneru commented on HDFS-12997:
---
Thanks [~ajayydv] for working on this.
# In {{DataStorage#doRollback(), }} there is a missing {}.
{code:java}
LOG.info("Rolling back storage directory .\n target LV = {}; target "
+ "CTime = {}", sd.getRoot(), nsInfo.getCTime(),
HdfsServerConstants.DATANODE_LAYOUT_VERSION);
{code}
# Nitpick: In {{DataStorage#removeDuplicateEntries()}}, line 1239, there is a
missing '.' (Just in case someone is running a log scanner with exact string
match :)).
> Move logging to slf4j in BlockPoolSliceStorage and Storage
> ---
>
> Key: HDFS-12997
> URL: https://issues.apache.org/jira/browse/HDFS-12997
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-12997.001.patch, HDFS-12997.002.patch,
> HDFS-12997.003.patch
>
>
> Move logging to slf4j in BlockPoolSliceStorage and Storage classes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12997) Move logging to slf4j in BlockPoolSliceStorage and Storage
[ https://issues.apache.org/jira/browse/HDFS-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340276#comment-16340276 ] Xiaoyu Yao commented on HDFS-12997: --- Thanks [~ajayydv] for working on this. Patch v3 looks good to me overall. Just few minor issues. +1 after that. DataStorage.java Line 936: this change does not correctly match the parameters. Please check and fix. Storage.java Line 1334: if (LOG.isDebugEnabled()) can be removed with slf4j as long as the number of parameters is less than 4 and there is no function call to get the value of the parameter. More details can be found here: https://www.slf4j.org/faq.html#logging_performance > Move logging to slf4j in BlockPoolSliceStorage and Storage > --- > > Key: HDFS-12997 > URL: https://issues.apache.org/jira/browse/HDFS-12997 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-12997.001.patch, HDFS-12997.002.patch, > HDFS-12997.003.patch > > > Move logging to slf4j in BlockPoolSliceStorage and Storage classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13059) Add pie chart in NN UI to show storage used
[ https://issues.apache.org/jira/browse/HDFS-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340256#comment-16340256 ] Ajay Kumar commented on HDFS-13059: --- [~aw] I am open to changing this to bar graph, do you have any specific ideas on how to handle concerns of multiple cluster or accessibility issue in that? > Add pie chart in NN UI to show storage used > --- > > Key: HDFS-13059 > URL: https://issues.apache.org/jira/browse/HDFS-13059 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13059.001.patch, HDFS-13059.002.patch, > HDFS-13059.003.patch, Screen Shot 2018-01-24 at 1.58.28 PM.png, Screen Shot > 2018-01-24 at 1.58.33 PM.png, Screen Shot 2018-01-24 at 3.04.17 PM.png, > fed-capacity.png > > > This jira proposes to add a pie chart in NN UI to show storage used by: > * DFS Used (Tooltip : "Storage currently used for DFS.") > * DFS available (Tooltip : "Storage available for DFS use.") > * Non DFS Used (Tooltip : "Storage allocated for DFS but currently" + > " used by Non DFS storage.") > Tooltip will help users better understand what these terms mean. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13044) RBF: Add a safe mode for the Router
[
https://issues.apache.org/jira/browse/HDFS-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340253#comment-16340253
]
Íñigo Goiri commented on HDFS-13044:
I added this new {{RouterSafeModeException}} and documented a little the
behavior.
> RBF: Add a safe mode for the Router
> ---
>
> Key: HDFS-13044
> URL: https://issues.apache.org/jira/browse/HDFS-13044
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-13004.000.patch, HDFS-13044.001.patch,
> HDFS-13044.002.patch
>
>
> When a Router cannot communicate with the State Store, it should enter into a
> safe mode that disallows certain operations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13044) RBF: Add a safe mode for the Router
[ https://issues.apache.org/jira/browse/HDFS-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13044: --- Attachment: HDFS-13044.002.patch > RBF: Add a safe mode for the Router > --- > > Key: HDFS-13044 > URL: https://issues.apache.org/jira/browse/HDFS-13044 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13004.000.patch, HDFS-13044.001.patch, > HDFS-13044.002.patch > > > When a Router cannot communicate with the State Store, it should enter into a > safe mode that disallows certain operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13044) RBF: Add a safe mode for the Router
[
https://issues.apache.org/jira/browse/HDFS-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340236#comment-16340236
]
Íñigo Goiri commented on HDFS-13044:
Thanks [~linyiqun], I think I tackled all your comments.
I left the one about defining the time to enter in safe mode as I think is more
intuitive than a factor (changed the name).
For the SafeMode one, I switched it but I need to test what is the behavior.
The goal here is that when we have a Router in safe mode, the client would
switch to another one.
With standby that's the behavior, but not sure about safe mode; I may create a
{{RouterSafeModeException}} which inherits from {{StandbyException}}.
> RBF: Add a safe mode for the Router
> ---
>
> Key: HDFS-13044
> URL: https://issues.apache.org/jira/browse/HDFS-13044
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-13004.000.patch, HDFS-13044.001.patch
>
>
> When a Router cannot communicate with the State Store, it should enter into a
> safe mode that disallows certain operations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13044) RBF: Add a safe mode for the Router
[ https://issues.apache.org/jira/browse/HDFS-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13044: --- Attachment: HDFS-13044.001.patch > RBF: Add a safe mode for the Router > --- > > Key: HDFS-13044 > URL: https://issues.apache.org/jira/browse/HDFS-13044 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13004.000.patch, HDFS-13044.001.patch > > > When a Router cannot communicate with the State Store, it should enter into a > safe mode that disallows certain operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336895#comment-16336895
]
Xiao Chen edited comment on HDFS-12528 at 1/25/18 10:29 PM:
Attaching a patch to do just #1, with enhanced unit tests from John to test
behavior for the default (disable for 10 minutes) and the never disable (0).
Reason didn't do #2 is I think the threshold is really a subjective thing, and
it feels to me having #1 would be sufficient for the requests so far from this
jira.
(fat-fingered to upload a wrong file. now it's the real patch 1)
was (Author: xiaochen):
Attaching a patch to do just #1, with enhanced unit tests from John to test
behavior for the default (disable for 10 minutes) and the never disable (0).
Reason didn't do #2 is I think the threshold is really a subjective thing, and
it feels to me having #1 would be sufficient for the requests so far from this
jira.
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HDFS-12528.000.patch, HDFS-12528.01.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340176#comment-16340176
]
Misha Dmitriev edited comment on HDFS-12051 at 1/25/18 10:29 PM:
-
Thank you for the review, [~manojg] See my responses inline below.
{{NameCache.java}}
* _line 97: {{cache = new byte[cacheSize][];}} Since this will take up a
contiguous space, we need to restrict the cache size to much lesser size than
your current MAX size of 1 << 30. Your thoughts?_
As you can see from line 78, the cache size is always set from the
configuration, which provides a reasonable default, which is much, much smaller
than 1<<30. It's up to the customer to increase this value if they need. If
they have a huge heap, like 120GB (I've already heard of users approaching
this!), then with 1<<30 it will result in an 8GB contiguous array. With a huge
heap already, it is nothing wrong, if they really need it. But, anyway, if they
decide to change this number, I think it's reasonable to expect them to have
some understanding of what they are doing.
_{{#cache}} is now following the {{open addressing}} model. Any reasons why you
moved to this model compared to your initial design?_
My own design for this cache has always been open addressing. The reason is
that this is the most economical model in terms of memory: it uses just one
pointer per cache entry, which is 8 bytes at most. If you use a cache with
collision chains, like in java.util.HashMap, then each entry requires a pointer
and a separate object (HashMap$Entry) This separate object takes at least 32
bytes, so you end up with at least 40 bytes per entry - five times more!
Now, for a real HashMap, that needs to hold potentially very large number of
objects, and needs to hold them all, the collision chain design may be
justified in some cases. But for our specialized fixed-size cache, that strives
to minimize its own memory overhead, the open addressing design is more
appropriate.
* _{{#put()}}_
** _line 118: the first time cache fill .. shouldn't it be a new byte array
name constructed from the passed in name? Why use the same caller passed in
name?_
The goal of this cache is to _avoid_ object duplication as much as possible. If
the caller gave us a name X for which we don't have an existing copy, just
remember X and return it. If on the next invocation they gave us Y and it turns
out to be the same as X, return X again, and Y will be effectively lost and
GCed soon.
* _With the {{open addressing}} model, when you overwrite the cache slot with
new names, there could be INodes which are already referring to this name and
are cut from the cache. Though their references are still valid, want to
understand why the preference given to new names compared to the old one._
The preference is given to new names simply because it's the lesser evil. We
already discussed this with [~yzhangal] in the past. First, obviously when a
cache entry is overwritten, the old INodes will just continue to refer to their
old names, i.e. no information is lost. Second, all our solution details stem
from the fact that we don't know in advance how many names we are going to
have, and how many of them will be duplicate. Thus we want to have a fixed-size
cache that will be guaranteed to not waste much memory if there is little
duplication, but will provide a benefit and will save a lot of memory if there
is considerable duplication.
Now, suppose we have a cache of size 3, and names come as follows: 'A', 'B',
'C', 'D', 'D', 'D', ... The cache would be full after the first 3 names. If
after that we don't override one of the entries to accomodate 'D', we will not
realize any savings from deduplicating all the subsequent 'D's. To be fair, if
this cache receives something like 'A', 'B', 'C', 'D', 'E', 'F', 'A', 'B', 'C',
'D', 'E', 'F' - then it just gets rewritten all the time and provides no
benefit. But in practice (and I have already implemented a similar cache in
several other projects), I've never observed such a pathology. With a
reasonable-size cache and real-life data, it always works.
* _I don't see any cache invalidation even when the INodes are removed. This
takes up memory. Though not huge, design wise its not clean to leave the cache
with stale values and incur cache lookup penalty in the future put()_
This cache by default takes just 16MB, which is 0.1% of 16GB, which is on the
smaller side of NN heap size spectrum. So any losses due to stale cache entries
are pretty negligible. Furthermore, the above-mentioned overwriting of cache
entries when new data is coming also helps to keep the cache reasonably "fresh".
* _{{#getSize()}} since there is no cache invalidation, and since this open
addressing model, the size returned is not right._
As the javadoc for this method explains, this method may return a slightly
[jira] [Updated] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-12528:
-
Attachment: HDFS-12528.01.patch
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HDFS-12528.000.patch, HDFS-12528.01.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12051:
--
Status: Patch Available (was: In Progress)
Addressed the most recent comments by [~yzhangal] and [~manojg]
> Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly
> those denoting file/directory names) to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch, HDFS-12051.07.patch, HDFS-12051.08.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code:java}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
>
[jira] [Updated] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-12528:
-
Attachment: (was: blocks_16_12_chars-20180123.png)
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HDFS-12528.000.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12051:
--
Attachment: HDFS-12051.08.patch
> Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly
> those denoting file/directory names) to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch, HDFS-12051.07.patch, HDFS-12051.08.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code:java}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
>
[jira] [Updated] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12051:
--
Status: In Progress (was: Patch Available)
> Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly
> those denoting file/directory names) to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch, HDFS-12051.07.patch, HDFS-12051.08.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code:java}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
>
[jira] [Updated] (HDFS-13061) SaslDataTransferClient#checkTrustAndSend should not trust a partially trusted channel
[ https://issues.apache.org/jira/browse/HDFS-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13061: -- Description: HDFS-5910 introduces encryption negotiation between client and server based on a customizable TrustedChannelResolver class. The TrustedChannelResolver is invoked on both client and server side. If the resolver indicates that the channel is trusted, then the data transfer will not be encrypted even if dfs.encrypt.data.transfer is set to true. SaslDataTransferClient#checkTrustAndSend ask the channel resolve whether the client and server address are trusted, respectively. It decides the channel is untrusted only if both client and server are not trusted to enforce encryption. *This ticket is opened to change it to not trust (and encrypt) if either client or server address are not trusted.* was: HDFS-5920 introduces encryption negotiation between client and server based on a customizable TrustedChannelResolver class. The TrustedChannelResolver is invoked on both client and server side. If the resolver indicates that the channel is trusted, then the data transfer will not be encrypted even if dfs.encrypt.data.transfer is set to true. SaslDataTransferClient#checkTrustAndSend ask the channel resolve whether the client and server address are trusted, respectively. It decides the channel is untrusted only if both client and server are not trusted to enforce encryption. *This ticket is opened to change it to not trust (and encrypt) if either client or server address are not trusted.* > SaslDataTransferClient#checkTrustAndSend should not trust a partially trusted > channel > - > > Key: HDFS-13061 > URL: https://issues.apache.org/jira/browse/HDFS-13061 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > > HDFS-5910 introduces encryption negotiation between client and server based > on a customizable TrustedChannelResolver class. The TrustedChannelResolver is > invoked on both client and server side. If the resolver indicates that the > channel is trusted, then the data transfer will not be encrypted even if > dfs.encrypt.data.transfer is set to true. > SaslDataTransferClient#checkTrustAndSend ask the channel resolve whether the > client and server address are trusted, respectively. It decides the channel > is untrusted only if both client and server are not trusted to enforce > encryption. *This ticket is opened to change it to not trust (and encrypt) if > either client or server address are not trusted.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13060) Adding a BlacklistBasedTrustedChannelResolver for TrustedChannelResolver
[ https://issues.apache.org/jira/browse/HDFS-13060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13060: -- Description: HDFS-5910 introduces encryption negotiation between client and server based on a customizable TrustedChannelResolver class. The TrustedChannelResolver is invoked on both client and server side. If the resolver indicates that the channel is trusted, then the data transfer will not be encrypted even if dfs.encrypt.data.transfer is set to true. The default trust channel resolver implementation returns false indicating that the channel is not trusted, which always enables encryption. HDFS-5910 also added a build-int whitelist based trust channel resolver. It allows you to put IP address/Network Mask of trusted client/server in whitelist files to skip encryption for certain traffics. This ticket is opened to add a blacklist based trust channel resolver for cases only certain machines (IPs) are untrusted without adding each trusted IP individually. was: HDFS-5920 introduces encryption negotiation between client and server based on a customizable TrustedChannelResolver class. The TrustedChannelResolver is invoked on both client and server side. If the resolver indicates that the channel is trusted, then the data transfer will not be encrypted even if dfs.encrypt.data.transfer is set to true. The default trust channel resolver implementation returns false indicating that the channel is not trusted, which always enables encryption. HDFS-5920 also added a build-int whitelist based trust channel resolver. It allows you to put IP address/Network Mask of trusted client/server in whitelist files to skip encryption for certain traffics. This ticket is opened to add a blacklist based trust channel resolver for cases only certain machines (IPs) are untrusted without adding each trusted IP individually. > Adding a BlacklistBasedTrustedChannelResolver for TrustedChannelResolver > > > Key: HDFS-13060 > URL: https://issues.apache.org/jira/browse/HDFS-13060 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > > HDFS-5910 introduces encryption negotiation between client and server based > on a customizable TrustedChannelResolver class. The TrustedChannelResolver is > invoked on both client and server side. If the resolver indicates that the > channel is trusted, then the data transfer will not be encrypted even if > dfs.encrypt.data.transfer is set to true. > The default trust channel resolver implementation returns false indicating > that the channel is not trusted, which always enables encryption. HDFS-5910 > also added a build-int whitelist based trust channel resolver. It allows you > to put IP address/Network Mask of trusted client/server in whitelist files to > skip encryption for certain traffics. > This ticket is opened to add a blacklist based trust channel resolver for > cases only certain machines (IPs) are untrusted without adding each trusted > IP individually. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340176#comment-16340176
]
Misha Dmitriev commented on HDFS-12051:
---
Thank you for the review, [~manojg] See my responses inline below.
{{NameCache.java}}
* _line 97: {{cache = new byte[cacheSize][];}} Since this will take up a
contiguous space, we need to restrict the cache size to much lesser size than
your current MAX size of 1 << 30. Your thoughts?_
As you can see from line 78, the cache size is always set from the
configuration, which provides a reasonable default, which is much, much smaller
than 1<<30. It's up to the customer to increase this value if they need. If
they have a huge heap, like 120GB (I've already heard of users approaching
this!), then with 1<<30 it will result in an 8GB contiguous array. With a huge
heap already, it is nothing wrong, if they really need it. But, anyway, if they
decide to change this number, I think it's reasonable to expect them to have
some understanding of what they are doing.
_{{#cache}} is now following the {{open addressing}} model. Any reasons why you
moved to this model compared to your initial design?_
My own design for this cache has always been open addressing. The reason is
that this is the most economical model in terms of memory: it uses just one
pointer per cache entry, which is 8 bytes at most. If you use a cache with
collision chains, like in java.util.HashMap, then each entry requires a pointer
and a separate object (HashMap$Entry) This separate object takes at least 32
bytes, so you end up with at least 40 bytes per entry - five times more!
Now, for a real HashMap, that needs to hold potentially very large number of
objects, and needs to hold them all, the collision chain design may be
justified in some cases. But for our specialized fixed-size cache, that strives
to minimize its own memory overhead, the open addressing design is more
appropriate.
* _{{#put()}}_
** _line 118: the first time cache fill .. shouldn't it be a new byte array
name constructed from the passed in name? Why use the same caller passed in
name?_
The goal of this cache is to _avoid_ object duplication as much as possible. If
the caller gave us a name X for which we don't have an existing copy, just
remember X and return it. If on the next invocation they gave us Y and it turns
out to be the same as X, return X again, and Y will be effectively lost and
GCed soon.
* _With the {{open addressing}} model, when you overwrite the cache slot with
new names, there could be INodes which are already referring to this name and
are cut from the cache. Though their references are still valid, want to
understand why the preference given to new names compared to the old one._
The preference is given to new names simply because it's the lesser evil. We
already discussed this with [~yzhangal] in the past. First, obviously when a
cache entry is overwritten, the old INodes will just continue to refer to their
old names, i.e. no information is lost. Second, all our solution details stem
from the fact that we don't know in advance how many names we are going to
have, and how many of them will be duplicate. Thus we want to have a fixed-size
cache that will be guaranteed to not waste much memory if there is little
duplication, but will provide a benefit and will save a lot of memory if there
is considerable duplication.
Now, suppose we have a cache of size 3, and names come as follows: 'A', 'B',
'C', 'D', 'D', 'D', ... The cache would be full after the first 3 names. If
after that we don't override one of the entries to accomodate 'D', we will not
realize any savings from deduplicating all the subsequent 'D's. To be fair, if
this cache receives something like 'A', 'B', 'C', 'D', 'E', 'F', 'A', 'B', 'C',
'D', 'E', 'F' - then it just gets rewritten all the time and provides no
benefit. But in practice (and I have already implemented a similar cache in
several other projects), I've never observed such a pathology. With a
reasonable-size cache and real-life data, it always works.
* _I don't see any cache invalidation even when the INodes are removed. This
takes up memory. Though not huge, design wise its not clean to leave the cache
with stale values and incur cache lookup penalty in the future put()_
This cache by default takes just 16MB, which is 0.1% of 16GB, which is on the
smaller side of NN heap size spectrum. So any losses due to stale cache entries
are pretty negligible. Furthermore, the above-mentioned overwriting of cache
entries when new data is coming also helps to keep the cache reasonably "fresh".
* _{{#getSize()}} since there is no cache invalidation, and since this open
addressing model, the size returned is not right._
As the javadoc for this method explains, this method may return a slightly
incorrect result because of races, and is intended to
[jira] [Commented] (HDFS-12522) Ozone: Remove the Priority Queues used in the Container State Manager
[
https://issues.apache.org/jira/browse/HDFS-12522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340162#comment-16340162
]
Anu Engineer commented on HDFS-12522:
-
[~xyao] and [~nandakumar131] Thanks for review comments, I have uploaded v3 of
the patch.
Here are the changes in the v3.
bq. ContainerStateMap.java Line:86 Extra “;”
Fixed.
bq. ContainerStateMap#updateState : As the name suggests, we should use it only
for updating the state of the container.
I wish we could get something as clean as this. Many times a state change
happens due to some variable change its state. So we have to update both the
state and value change in an atomic fashion.
bq. line numbers were 427 - 429
Thanks for letting me know.
bq. SCMException.java: NIT: Line 117 unncessary change.
Good catch, fixed.
bq. BlockManagerImpl.java Line 270: the udpate of allocatedbytes looks good to
me. I notice that we have a TODO in 421 to reclaim the deleted block space from
container. Please ensure we have a open JIRA on this to fix this.
Filed HDFS-13067.
bq. ContainerMapping.java Line 408, # of parameter does not match the # of {}
in the LOG.error statement.
Thank you for catching that. I fixed both the log lines that were faulty.
> Ozone: Remove the Priority Queues used in the Container State Manager
> -
>
> Key: HDFS-12522
> URL: https://issues.apache.org/jira/browse/HDFS-12522
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>Priority: Major
> Attachments: HDFS-12522-HDFS-7240.001.patch,
> HDFS-12522-HDFS-7240.002.patch, HDFS-12522-HDFS-7240.003.patch
>
>
> During code review of HDFS-12387, it was suggested that we remove the
> priority queues that was used in ContainerStateManager. This JIRA tracks that
> issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12522) Ozone: Remove the Priority Queues used in the Container State Manager
[ https://issues.apache.org/jira/browse/HDFS-12522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12522: Attachment: HDFS-12522-HDFS-7240.003.patch > Ozone: Remove the Priority Queues used in the Container State Manager > - > > Key: HDFS-12522 > URL: https://issues.apache.org/jira/browse/HDFS-12522 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > Attachments: HDFS-12522-HDFS-7240.001.patch, > HDFS-12522-HDFS-7240.002.patch, HDFS-12522-HDFS-7240.003.patch > > > During code review of HDFS-12387, it was suggested that we remove the > priority queues that was used in ContainerStateManager. This JIRA tracks that > issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13057) [SPS]: Revisit configurations to make SPS service modes internal/external/none
[
https://issues.apache.org/jira/browse/HDFS-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340148#comment-16340148
]
genericqa commented on HDFS-13057:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
10s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} HDFS-10285 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 1s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
891 unchanged - 0 fixed = 892 total (was 891) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
36s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}129m 11s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}197m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.namenode.TestStoragePolicySatisfierWithHA |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| | hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13057 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340110#comment-16340110
]
genericqa commented on HDFS-13022:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
3s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
20s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
14s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 10s{color} | {color:orange} root: The patch generated 10 new + 0 unchanged -
1 fixed = 10 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 12m
52s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
53s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
12s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
55s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}146m 22s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
47s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}280m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Write to static field
org.apache.hadoop.cblock.kubernetes.DynamicProvisioner.running from instance
method org.apache.hadoop.cblock.kubernetes.DynamicProvisioner.stop() At
DynamicProvisioner.java:from instance method
org.apache.hadoop.cblock.kubernetes.DynamicProvisioner.stop() At
DynamicProvisioner.java:[line 193] |
| Failed junit tests |
[jira] [Commented] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[
https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340102#comment-16340102
]
Virajith Jalaparti commented on HDFS-12942:
---
Thanks for the updated patch [~ajayydv]. A couple of issues with patch v5 that
are causing the failed tests.
{code:java}
+ FsVolumeImpl volume = (FsVolumeImpl) newReplicaInfo.getVolume();
+ volume.incDfsUsedAndNumBlocks(bpid, newReplicaInfo.getBytesOnDisk())
{code}
This shouldn't be added to {{finalizeReplica}}. This should be called in
{{finalizeNewReplica}} if {{finalizeReplica}} succeeds.
Also, after the deletions in the catch block in {{finalizeNewReplica}}, the
exception has to be re-thrown.
> Synchronization issue in FSDataSetImpl#moveBlock
>
>
> Key: HDFS-12942
> URL: https://issues.apache.org/jira/browse/HDFS-12942
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch,
> HDFS-12942.003.patch, HDFS-12942.004.patch, HDFS-12942.005.patch
>
>
> FSDataSetImpl#moveBlock works in following following 3 steps:
> # first creates a new replicaInfo object
> # calls finalizeReplica to finalize it.
> # Calls removeOldReplica to remove oldReplica.
> A client can potentially append to the old replica between step 1 and 2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12942: -- Attachment: (was: HDFS-12942.006.patch) > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch, > HDFS-12942.003.patch, HDFS-12942.004.patch, HDFS-12942.005.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12942: -- Attachment: HDFS-12942.006.patch > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch, > HDFS-12942.003.patch, HDFS-12942.004.patch, HDFS-12942.005.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13067) Ozone: Update the allocatedBlock size in SCM when delete blocks happen.
[ https://issues.apache.org/jira/browse/HDFS-13067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-13067: Issue Type: Sub-task (was: Bug) Parent: HDFS-7240 > Ozone: Update the allocatedBlock size in SCM when delete blocks happen. > --- > > Key: HDFS-13067 > URL: https://issues.apache.org/jira/browse/HDFS-13067 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Priority: Trivial > > We rely on Container Reports to understand the actually allocated size of a > container. We also maintain another counter that keeps track of the logical > allocations. That is the number of blocks allocated in the container, while > this number is used only to queue containers for closing it might be a good > idea to make sure that this number is updated when a delete block operation > is performed, Simply because we have the data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13049) RBF: Inconsistent Router OPTS config in branch-2 and branch-3
[ https://issues.apache.org/jira/browse/HDFS-13049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated HDFS-13049: --- Resolution: Fixed Fix Version/s: 3.0.1 2.9.1 2.10.0 3.1.0 Status: Resolved (was: Patch Available) Thanks [~elgoiri] for the reviews. Committed to trunk, branch-3.0, branch-2, branch-2.9. > RBF: Inconsistent Router OPTS config in branch-2 and branch-3 > - > > Key: HDFS-13049 > URL: https://issues.apache.org/jira/browse/HDFS-13049 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Minor > Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1 > > Attachments: HDFS-13049-branch-2.001.patch, > HDFS-13049-branch-2.002.patch, HDFS-13049.001.patch, HDFS-13049.002.patch > > > For router's OPS config: > * In trunk/branch-3, the cmd looks for (command)_(subcommad)_OPTS --> > HDFS_DFSROUTER_OPTS / HADOOP_DFSROUTER_OPTS. > * In branch-2, the cmd looks for HADOOP_ROUTER_OPTS. > Also, it would be better to add commented corresponding config in > hadoop-env.sh, for better user visibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13067) Ozone: Update the allocatedBlock size in SCM when delete blocks happen.
Anu Engineer created HDFS-13067: --- Summary: Ozone: Update the allocatedBlock size in SCM when delete blocks happen. Key: HDFS-13067 URL: https://issues.apache.org/jira/browse/HDFS-13067 Project: Hadoop HDFS Issue Type: Bug Reporter: Anu Engineer We rely on Container Reports to understand the actually allocated size of a container. We also maintain another counter that keeps track of the logical allocations. That is the number of blocks allocated in the container, while this number is used only to queue containers for closing it might be a good idea to make sure that this number is updated when a delete block operation is performed, Simply because we have the data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13051) dead lock occurs when rolleditlog rpc call happen and editPendingQ is full
[ https://issues.apache.org/jira/browse/HDFS-13051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339846#comment-16339846 ] Daryn Sharp commented on HDFS-13051: Almost have a very small fix ready, just making sure there is a tight test case that specifically exercises the code paths. Instead of current tests that just spin for awhile and assume everything worked... > dead lock occurs when rolleditlog rpc call happen and editPendingQ is full > -- > > Key: HDFS-13051 > URL: https://issues.apache.org/jira/browse/HDFS-13051 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.5 >Reporter: zhangwei >Priority: Major > Labels: AsyncEditlog, deadlock > > when doing rolleditlog it acquires fs write lock，then acquire FSEditLogAsync > lock object，and write 3 EDIT(the second one override logEdit method and > return true) > in extremely case，when FSEditLogAsync's logSync is very > slow，editPendingQ(default size 4096)is full，it case IPC thread can not offer > edit object into editPendingQ when doing rolleditlog，it block on editPendingQ > .put method，however it does't release FSEditLogAsync object lock, and > edit.logEdit method in FSEditLogAsync.run thread can never acquire > FSEditLogAsync object lock, it case dead lock > stack trace like below > "Thread[Thread-44528,5,main]" #130093 daemon prio=5 os_prio=0 > tid=0x02377000 nid=0x13fda waiting on condition [0x7fb3297de000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7fbd3cb96f58> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.enqueueEdit(FSEditLogAsync.java:156) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.logEdit(FSEditLogAsync.java:118) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLog.logCancelDelegationToken(FSEditLog.java:1008) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.logExpireDelegationToken(FSNamesystem.java:7635) > at > org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSecretManager.logExpireToken(DelegationTokenSecretManager.java:395) > - locked <0x7fbd3cbae500> (a java.lang.Object) > at > org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSecretManager.logExpireToken(DelegationTokenSecretManager.java:62) > at > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.removeExpiredToken(AbstractDelegationTokenSecretManager.java:604) > at > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.access$400(AbstractDelegationTokenSecretManager.java:54) > at > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$ExpiredTokenRemover.run(AbstractDelegationTokenSecretManager.java:656) > at java.lang.Thread.run(Thread.java:745) > "FSEditLogAsync" #130072 daemon prio=5 os_prio=0 tid=0x0715b800 > nid=0x13fbf waiting for monitor entry [0x7fb32c51a000] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLog.doEditTransaction(FSEditLog.java:443) > - waiting to lock <*0x7fbcbc131000*> (a > org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync$Edit.logEdit(FSEditLogAsync.java:233) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSEditLogAsync.java:177) > at java.lang.Thread.run(Thread.java:745) > "IPC Server handler 47 on 53310" #337 daemon prio=5 os_prio=0 > tid=0x7fe659d46000 nid=0x4c62 waiting on condition [0x7fb32fe52000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7fbd3cb96f58> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.enqueueEdit(FSEditLogAsync.java:156) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.logEdit(FSEditLogAsync.java:118) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1251) > - locked
[jira] [Assigned] (HDFS-13051) dead lock occurs when rolleditlog rpc call happen and editPendingQ is full
[ https://issues.apache.org/jira/browse/HDFS-13051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp reassigned HDFS-13051: -- Assignee: Daryn Sharp > dead lock occurs when rolleditlog rpc call happen and editPendingQ is full > -- > > Key: HDFS-13051 > URL: https://issues.apache.org/jira/browse/HDFS-13051 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.5 >Reporter: zhangwei >Assignee: Daryn Sharp >Priority: Major > Labels: AsyncEditlog, deadlock > > when doing rolleditlog it acquires fs write lock，then acquire FSEditLogAsync > lock object，and write 3 EDIT(the second one override logEdit method and > return true) > in extremely case，when FSEditLogAsync's logSync is very > slow，editPendingQ(default size 4096)is full，it case IPC thread can not offer > edit object into editPendingQ when doing rolleditlog，it block on editPendingQ > .put method，however it does't release FSEditLogAsync object lock, and > edit.logEdit method in FSEditLogAsync.run thread can never acquire > FSEditLogAsync object lock, it case dead lock > stack trace like below > "Thread[Thread-44528,5,main]" #130093 daemon prio=5 os_prio=0 > tid=0x02377000 nid=0x13fda waiting on condition [0x7fb3297de000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7fbd3cb96f58> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.enqueueEdit(FSEditLogAsync.java:156) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.logEdit(FSEditLogAsync.java:118) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLog.logCancelDelegationToken(FSEditLog.java:1008) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.logExpireDelegationToken(FSNamesystem.java:7635) > at > org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSecretManager.logExpireToken(DelegationTokenSecretManager.java:395) > - locked <0x7fbd3cbae500> (a java.lang.Object) > at > org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSecretManager.logExpireToken(DelegationTokenSecretManager.java:62) > at > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.removeExpiredToken(AbstractDelegationTokenSecretManager.java:604) > at > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.access$400(AbstractDelegationTokenSecretManager.java:54) > at > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$ExpiredTokenRemover.run(AbstractDelegationTokenSecretManager.java:656) > at java.lang.Thread.run(Thread.java:745) > "FSEditLogAsync" #130072 daemon prio=5 os_prio=0 tid=0x0715b800 > nid=0x13fbf waiting for monitor entry [0x7fb32c51a000] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLog.doEditTransaction(FSEditLog.java:443) > - waiting to lock <*0x7fbcbc131000*> (a > org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync$Edit.logEdit(FSEditLogAsync.java:233) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSEditLogAsync.java:177) > at java.lang.Thread.run(Thread.java:745) > "IPC Server handler 47 on 53310" #337 daemon prio=5 os_prio=0 > tid=0x7fe659d46000 nid=0x4c62 waiting on condition [0x7fb32fe52000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7fbd3cb96f58> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.enqueueEdit(FSEditLogAsync.java:156) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.logEdit(FSEditLogAsync.java:118) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1251) > - locked <*0x7fbcbc131000*> (a > org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:1190) > - locked
[jira] [Commented] (HDFS-13049) RBF: Inconsistent Router OPTS config in branch-2 and branch-3
[ https://issues.apache.org/jira/browse/HDFS-13049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339814#comment-16339814 ] Hudson commented on HDFS-13049: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13558 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13558/]) HDFS-13049. RBF: Inconsistent Router OPTS config in branch-2 and (weiy: rev 0c139d5bcfbcd62fc69111ee6204926c57d57bf1) * (edit) hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.cmd * (edit) hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh > RBF: Inconsistent Router OPTS config in branch-2 and branch-3 > - > > Key: HDFS-13049 > URL: https://issues.apache.org/jira/browse/HDFS-13049 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Minor > Attachments: HDFS-13049-branch-2.001.patch, > HDFS-13049-branch-2.002.patch, HDFS-13049.001.patch, HDFS-13049.002.patch > > > For router's OPS config: > * In trunk/branch-3, the cmd looks for (command)_(subcommad)_OPTS --> > HDFS_DFSROUTER_OPTS / HADOOP_DFSROUTER_OPTS. > * In branch-2, the cmd looks for HADOOP_ROUTER_OPTS. > Also, it would be better to add commented corresponding config in > hadoop-env.sh, for better user visibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12897) Path not found when we get the ec policy for a .snapshot dir
[
https://issues.apache.org/jira/browse/HDFS-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339796#comment-16339796
]
genericqa commented on HDFS-12897:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}124m 26s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}172m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12897 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905833/HDFS-12897.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux bfb52449b016 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 82cc6f6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22816/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22816/testReport/ |
| Max. process+thread count | 3425 (vs. ulimit of 5000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/22816/console |
| Powered by | Apache Yetus
[jira] [Updated] (HDFS-12812) Ozone: dozone: initialize scm and ksm directory on cluster startup
[ https://issues.apache.org/jira/browse/HDFS-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-12812: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Status: Resolved (was: Patch Available) Thanks [~elek] for the contribution and all for the reviews, I've committed the patch to the feature branch. > Ozone: dozone: initialize scm and ksm directory on cluster startup > -- > > Key: HDFS-12812 > URL: https://issues.apache.org/jira/browse/HDFS-12812 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: HDFS-7240 > > Attachments: HADOOP-12812-HDFS-7240.001.patch, > HDFS-12812-HDFS-7240.002.patch > > > HDFS-12739 fixed the scm, but after the patch it couldn't be started without > a separated `hdfs scm -init` any more. Unfortunatelly it breaks the > docker-compose functionality. > This is handled int the starter script of the base runner docker image for > namenode. I also fixed this in the runner docker image > (https://github.com/elek/hadoop/commit/b347eb4bfca37d84dbcdd8c4bf353219d876a9b7) > will upload the improved patch to the HADOOP-14898. > In this patch I just add a new environment variable to init the scm if the > version file doesn't exist. > UPDATE: the patch also contains envrionment variable to initialize the ksm. > To test: > Do a full build and go to the dev-support/compose/ozone: > ``` > docker-compose pull > docker-compose up -d > ``` > Note: the pull is important as the new docker image -- which could handle the > new environment variable -- should be used -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12812) Ozone: dozone: initialize scm and ksm directory on cluster startup
[ https://issues.apache.org/jira/browse/HDFS-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339789#comment-16339789 ] Xiaoyu Yao commented on HDFS-12812: --- Thanks [~elek] for fixing this. Patch looks good to me, +1. I've tested it locally and ksm/scm/nn/dn work as expected. I will commit it shortly. > Ozone: dozone: initialize scm and ksm directory on cluster startup > -- > > Key: HDFS-12812 > URL: https://issues.apache.org/jira/browse/HDFS-12812 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-12812-HDFS-7240.001.patch, > HDFS-12812-HDFS-7240.002.patch > > > HDFS-12739 fixed the scm, but after the patch it couldn't be started without > a separated `hdfs scm -init` any more. Unfortunatelly it breaks the > docker-compose functionality. > This is handled int the starter script of the base runner docker image for > namenode. I also fixed this in the runner docker image > (https://github.com/elek/hadoop/commit/b347eb4bfca37d84dbcdd8c4bf353219d876a9b7) > will upload the improved patch to the HADOOP-14898. > In this patch I just add a new environment variable to init the scm if the > version file doesn't exist. > UPDATE: the patch also contains envrionment variable to initialize the ksm. > To test: > Do a full build and go to the dev-support/compose/ozone: > ``` > docker-compose pull > docker-compose up -d > ``` > Note: the pull is important as the new docker image -- which could handle the > new environment variable -- should be used -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13053) Track time to process packet in Datanode
[
https://issues.apache.org/jira/browse/HDFS-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339716#comment-16339716
]
Hanisha Koneru commented on HDFS-13053:
---
Say the DN write pipeline is {{Client->DN1->DN2->DN3.}} IIUC, the packet
processing time at {{DN2}} would include write latency of forwarding the packet
to {{DN3}} also. And say {{DN3}} was slow in accepting packets. Then the
reported processing time of DN2 would be skewed. Please correct me if I am
wrong.
> Track time to process packet in Datanode
>
>
> Key: HDFS-13053
> URL: https://issues.apache.org/jira/browse/HDFS-13053
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Íñigo Goiri
>Assignee: Pulkit Misra
>Priority: Minor
> Attachments: HDFS-13053.000.patch
>
>
> We should track the time that each datanode takes to process a packet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12868) Ozone: Service Discovery API
[
https://issues.apache.org/jira/browse/HDFS-12868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-12868:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HFDS-7240
Status: Resolved (was: Patch Available)
Thanks [~nandakumar131] for the contribution. I've committed the patch to the
feature branch.
> Ozone: Service Discovery API
>
>
> Key: HDFS-12868
> URL: https://issues.apache.org/jira/browse/HDFS-12868
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: HFDS-7240
>
> Attachments: HDFS-12868-HDFS-7240.000.patch,
> HDFS-12868-HDFS-7240.001.patch, HDFS-12868-HDFS-7240.002.patch
>
>
> Currently if a client wants to connect to Ozone cluster we need multiple
> properties to be configured in the client.
> For RPC based connection we need
> {{ozone.ksm.address}}
> {{ozone.scm.client.address}}
> and the ports if something other than default is configured.
> For REST based connection
> {{ozone.rest.servers}}
> and port if something other than default is configured.
> With the introduction of Service Discovery API the client should be able to
> discover all the configurations needed for the connection. Service discovery
> calls will be handled by KSM, at the client side, we only need to configure
> {{ozone.ksm.address}}. The client should first connect to KSM and get all the
> required configurations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12868) Ozone: Service Discovery API
[
https://issues.apache.org/jira/browse/HDFS-12868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339663#comment-16339663
]
Xiaoyu Yao commented on HDFS-12868:
---
Thanks [~nandakumar131] for the update. Latest patch looks good to me, +1.
I tried the failed Jenkins tests and can't repro the failures. I will commit it
shortly.
> Ozone: Service Discovery API
>
>
> Key: HDFS-12868
> URL: https://issues.apache.org/jira/browse/HDFS-12868
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-12868-HDFS-7240.000.patch,
> HDFS-12868-HDFS-7240.001.patch, HDFS-12868-HDFS-7240.002.patch
>
>
> Currently if a client wants to connect to Ozone cluster we need multiple
> properties to be configured in the client.
> For RPC based connection we need
> {{ozone.ksm.address}}
> {{ozone.scm.client.address}}
> and the ports if something other than default is configured.
> For REST based connection
> {{ozone.rest.servers}}
> and port if something other than default is configured.
> With the introduction of Service Discovery API the client should be able to
> discover all the configurations needed for the connection. Service discovery
> calls will be handled by KSM, at the client side, we only need to configure
> {{ozone.ksm.address}}. The client should first connect to KSM and get all the
> required configurations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13057) [SPS]: Revisit configurations to make SPS service modes internal/external/none
[
https://issues.apache.org/jira/browse/HDFS-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339659#comment-16339659
]
Uma Maheswara Rao G commented on HDFS-13057:
Thank you [~rakeshr] for the patch.
Patch mostly look good, I have few minor comments though.
1.
{code:java}
+ if (childrenCount <= 0) {
+ service.addAllFileIdsToProcess(inodeId, new ArrayList<>(), true);
+ } else {
+ service.markScanCompletedForPath(inodeId);
+ }{code}
Could you add doc about about condition? May be a TODO to improve the condition
as well.
2.
{code:java}
+ // testMaxRetryForFailedBlock{code}
Can you remove this from TestExternalStoragePolicySatisfier
3. IsSatisfierRunningCommand: Probably usage should say, this is mainly tells
whether SPS running inside NN. I agree command itself self descriptive, it may
be worth clarifying in usage as well to be in sync.
4.
{code:java}
+ Administrator can dynamically change the StoragePolicySatisfier mode by using
reconfiguration option.
Dynamic enabling/disabling option can be achieved in the following way.{code}
I feel we should say change modes, instead of enable/disable
5. internal service to NN —> internal service in NN
6.
{code:java}
if (childrenCount <= 0) {
+ service.addAllFileIdsToProcess(inodeId, new ArrayList<>(), true);
+ }{code}
Can we put log here saying empty directory?
> [SPS]: Revisit configurations to make SPS service modes internal/external/none
> --
>
> Key: HDFS-13057
> URL: https://issues.apache.org/jira/browse/HDFS-13057
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Blocker
> Attachments: HDFS-13057-HDFS-10285-00.patch,
> HDFS-13057-HDFS-10285-01.patch
>
>
> This task is to revisit the configurations to make SPS service modes -
> {{internal/external/none}}
> - {{internal}} : represents SPS service will be running with NN
> - {{external}}: represents SPS service will be running outside NN
> - {{none}}: represents the SPS service is completely disabled and zero cost
> to the system.
> Proposed configuration {{dfs.storage.policy.satisfier.mode}} item in
> hdfs-site.xml file and value will be string. The mode can be changed via
> {{reconfig}} command.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13057) [SPS]: Revisit configurations to make SPS service modes internal/external/none
[
https://issues.apache.org/jira/browse/HDFS-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-13057:
Target Version/s: HDFS-10285
Status: Patch Available (was: Open)
Attached patch to support the modes. Please review, thanks!
> [SPS]: Revisit configurations to make SPS service modes internal/external/none
> --
>
> Key: HDFS-13057
> URL: https://issues.apache.org/jira/browse/HDFS-13057
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Blocker
> Attachments: HDFS-13057-HDFS-10285-00.patch,
> HDFS-13057-HDFS-10285-01.patch
>
>
> This task is to revisit the configurations to make SPS service modes -
> {{internal/external/none}}
> - {{internal}} : represents SPS service will be running with NN
> - {{external}}: represents SPS service will be running outside NN
> - {{none}}: represents the SPS service is completely disabled and zero cost
> to the system.
> Proposed configuration {{dfs.storage.policy.satisfier.mode}} item in
> hdfs-site.xml file and value will be string. The mode can be changed via
> {{reconfig}} command.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13057) [SPS]: Revisit configurations to make SPS service modes internal/external/none
[
https://issues.apache.org/jira/browse/HDFS-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-13057:
Attachment: HDFS-13057-HDFS-10285-01.patch
> [SPS]: Revisit configurations to make SPS service modes internal/external/none
> --
>
> Key: HDFS-13057
> URL: https://issues.apache.org/jira/browse/HDFS-13057
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Blocker
> Attachments: HDFS-13057-HDFS-10285-00.patch,
> HDFS-13057-HDFS-10285-01.patch
>
>
> This task is to revisit the configurations to make SPS service modes -
> {{internal/external/none}}
> - {{internal}} : represents SPS service will be running with NN
> - {{external}}: represents SPS service will be running outside NN
> - {{none}}: represents the SPS service is completely disabled and zero cost
> to the system.
> Proposed configuration {{dfs.storage.policy.satisfier.mode}} item in
> hdfs-site.xml file and value will be string. The mode can be changed via
> {{reconfig}} command.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13023) Journal Sync does not work on a secure cluster
[
https://issues.apache.org/jira/browse/HDFS-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339598#comment-16339598
]
Bharat Viswanadham commented on HDFS-13023:
---
Thank You [~hanishakoneru] for review and committing the changes.
> Journal Sync does not work on a secure cluster
> --
>
> Key: HDFS-13023
> URL: https://issues.apache.org/jira/browse/HDFS-13023
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Namit Maheshwari
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13023.00.patch, HDFS-13023.01.patch,
> HDFS-13023.02.patch, HDFS-13023.03.patch
>
>
> Fails with the following exception.
> {code}
> 2018-01-10 01:15:40,517 INFO server.JournalNodeSyncer
> (JournalNodeSyncer.java:syncWithJournalAtIndex(235)) - Syncing Journal
> /0.0.0.0:8485 with xxx, journal id: mycluster
> 2018-01-10 01:15:40,583 ERROR server.JournalNodeSyncer
> (JournalNodeSyncer.java:syncWithJournalAtIndex(259)) - Could not sync with
> Journal at xxx/xxx:8485
> com.google.protobuf.ServiceException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for
> protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol:
> this service is only accessible by nn/x...@example.com
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:242)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy16.getEditLogManifest(Unknown Source)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:254)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:230)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:190)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User nn/xxx (auth:PROXY) via jn/xxx (auth:KERBEROS) is not authorized for
> protocol interface org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol:
> this service is only accessible by nn/xxx
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> ... 6 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13059) Add pie chart in NN UI to show storage used
[ https://issues.apache.org/jira/browse/HDFS-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339580#comment-16339580 ] Allen Wittenauer commented on HDFS-13059: - bq. Our UI's are stateless and are not suitable for comparing any data point over period of time. I never said that the UI should show historical data[*]. I'm saying that people will generate their own form of historical data via the UI. I've lost track of how many conference presentations I've seen with screenshots of the NN UI. At this point, I'm bowing out of the topic. I've given my suggestion. [*] - it could, however, given the metrics system does have a level of it available. It again highlights why a bar chart or a stacked column chart would be significantly better. But that's a different set of improvements altogether. > Add pie chart in NN UI to show storage used > --- > > Key: HDFS-13059 > URL: https://issues.apache.org/jira/browse/HDFS-13059 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13059.001.patch, HDFS-13059.002.patch, > HDFS-13059.003.patch, Screen Shot 2018-01-24 at 1.58.28 PM.png, Screen Shot > 2018-01-24 at 1.58.33 PM.png, Screen Shot 2018-01-24 at 3.04.17 PM.png, > fed-capacity.png > > > This jira proposes to add a pie chart in NN UI to show storage used by: > * DFS Used (Tooltip : "Storage currently used for DFS.") > * DFS available (Tooltip : "Storage available for DFS use.") > * Non DFS Used (Tooltip : "Storage allocated for DFS but currently" + > " used by Non DFS storage.") > Tooltip will help users better understand what these terms mean. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13057) [SPS]: Revisit configurations to make SPS service modes internal/external/none
[
https://issues.apache.org/jira/browse/HDFS-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-13057:
Description:
This task is to revisit the configurations to make SPS service modes -
{{internal/external/none}}
- {{internal}} : represents SPS service will be running with NN
- {{external}}: represents SPS service will be running outside NN
- {{none}}: represents the SPS service is completely disabled and zero cost to
the system.
Proposed configuration {{dfs.storage.policy.satisfier.mode}} item in
hdfs-site.xml file and value will be string. The mode can be changed via
{{reconfig}} command.
was:
This task is to revisit the configurations to make SPS service modes -
{{internal/external/none}}
- {{internal}} : represents SPS service should be running with NN
- {{external}}: represents SPS service will be running outside NN
- {{none}}: represents the SPS service is completely disabled and zero cost to
the system.
Proposed configuration {{dfs.storage.policy.satisfier.running.mode}} item in
hdfs-site.xml file and value will be string. The mode can be changed via
{{reconfig}} command.
> [SPS]: Revisit configurations to make SPS service modes internal/external/none
> --
>
> Key: HDFS-13057
> URL: https://issues.apache.org/jira/browse/HDFS-13057
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Blocker
> Attachments: HDFS-13057-HDFS-10285-00.patch
>
>
> This task is to revisit the configurations to make SPS service modes -
> {{internal/external/none}}
> - {{internal}} : represents SPS service will be running with NN
> - {{external}}: represents SPS service will be running outside NN
> - {{none}}: represents the SPS service is completely disabled and zero cost
> to the system.
> Proposed configuration {{dfs.storage.policy.satisfier.mode}} item in
> hdfs-site.xml file and value will be string. The mode can be changed via
> {{reconfig}} command.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13065) TestErasureCodingMultipleRacks#testSkewedRack3 is failing
[
https://issues.apache.org/jira/browse/HDFS-13065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339543#comment-16339543
]
genericqa commented on HDFS-13065:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 8m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}118m 49s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13065 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907703/HDFS-13065.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 6ef1b4765c45 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 82cc6f6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/22811/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22811/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22811/testReport/ |
| Max. process+thread count | 3077 (vs. ulimit of 5000) |
|
[jira] [Comment Edited] (HDFS-13059) Add pie chart in NN UI to show storage used
[
https://issues.apache.org/jira/browse/HDFS-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339525#comment-16339525
]
Ajay Kumar edited comment on HDFS-13059 at 1/25/18 5:32 PM:
[~elgoiri],[~aw] thanks for your comments.
{quote}
When talking about storage there are always two extra data points: time and
growth. Administrators are going to use this pie chart inpresentations. Then
use it again in 6 months. It ends being a horrible comparison because unless
the usage of the cluster hasn't changed dramatically, it doesn't really convey
much information. {quote}
Our UI's are stateless and are not suitable for comparing any data point over
period of time. People usually rely on jmx data to do comparison over
historical data.
{quote} Given that the tool tip is required really underscores this
point.{quote}
I have seen people working on hadoop from years but still getting confused b/w
what DFS used and Non DFS used means in our summary table. Tooltip tries to
address that.
{quote}Additionally, if I have more than one cluster, I'm going to pull up both
NN UIs and look at both charts simultaneously. Again, this doesn't really
convey much information other than maybe the usage patterns are similar. It
doesn't convey an actual size.{quote}
Yes, but this is current behavior(how our current UI is) and this change is
not intended to address that. May be we can address it separately.
{quote} As an alternative, I think I'd much rather have a bar chart where some
actual numeric information can also be provided without requiring tool tips.
The units will also give a much better sense of quantity. Comparing static bar
charts over time is also significantly easier.{quote}
Personally, i think in this case we are showing composition of DFS allocation,
so pie chart is well suited. Concrete numbers are already there in Summary and
bar chart will show them numerically relative to each other. Also we don't have
the dimension of time in jmx data. So, benefit of bar graph to show data
changing over time is not applicable in this case.
I am open to changing this if others feel that bar graph will serve the purpose
better.
was (Author: ajayydv):
[~elgoiri],[~aw] thanks for your comments.
>When talking about storage there are always two extra data points: time and
>growth. Administrators are going to use this pie chart in
>presentations. Then use it again in 6 months. It ends being a horrible
>comparison because unless the usage of the cluster hasn't
>changed dramatically, it doesn't really convey much information.
Our UI's are stateless and are not suitable for comparing any data point over
period of time. People usually rely on jmx data to do comparison over
historical data.
> Given that the tool tip is required really underscores this point.
I have seen people working on hadoop from years but still getting confused b/w
what DFS used and Non DFS used means in our summary table. Tooltip tries to
address that.
>Additionally, if I have more than one cluster, I'm going to pull up both NN
>UIs and look at both charts simultaneously. Again, this doesn't
>really convey much information other than maybe the usage patterns are
>similar. It doesn't convey an actual size.
Yes, but this is current behavior(how our current UI is) and this change is
not intended to address that. May be we can address it separately.
>As an alternative, I think I'd much rather have a bar chart where some actual
>numeric information can also be provided without
>requiring tool tips. The units will also give a much better sense of quantity.
>Comparing static bar charts over time is also significantly easier.
Personally i think we are showing composition of DFS allocation, so pie chart
is well suited in this case. Concrete numbers are already there in Summary and
bar chart will show them numerically relative to each other. Also we don't have
the dimension of time in jmx data. So, benefit of bar graph to show data
changing over time is not applicable in this case.
I am open to changing this if others feel that bar graph will serve the purpose
better.
> Add pie chart in NN UI to show storage used
> ---
>
> Key: HDFS-13059
> URL: https://issues.apache.org/jira/browse/HDFS-13059
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-13059.001.patch, HDFS-13059.002.patch,
> HDFS-13059.003.patch, Screen Shot 2018-01-24 at 1.58.28 PM.png, Screen Shot
> 2018-01-24 at 1.58.33 PM.png, Screen Shot 2018-01-24 at 3.04.17 PM.png,
> fed-capacity.png
>
>
> This jira proposes to add a pie chart in NN UI to show storage used by:
> * DFS Used (Tooltip : "Storage currently used for DFS.")
> * DFS available (Tooltip :
[jira] [Comment Edited] (HDFS-13059) Add pie chart in NN UI to show storage used
[ https://issues.apache.org/jira/browse/HDFS-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339525#comment-16339525 ] Ajay Kumar edited comment on HDFS-13059 at 1/25/18 5:28 PM: [~elgoiri],[~aw] thanks for your comments. >When talking about storage there are always two extra data points: time and >growth. Administrators are going to use this pie chart in >presentations. Then use it again in 6 months. It ends being a horrible >comparison because unless the usage of the cluster hasn't >changed dramatically, it doesn't really convey much information. Our UI's are stateless and are not suitable for comparing any data point over period of time. People usually rely on jmx data to do comparison over historical data. > Given that the tool tip is required really underscores this point. I have seen people working on hadoop from years but still getting confused b/w what DFS used and Non DFS used means in our summary table. Tooltip tries to address that. >Additionally, if I have more than one cluster, I'm going to pull up both NN >UIs and look at both charts simultaneously. Again, this doesn't >really convey much information other than maybe the usage patterns are >similar. It doesn't convey an actual size. Yes, but this is current behavior(how our current UI is) and this change is not intended to address that. May be we can address it separately. >As an alternative, I think I'd much rather have a bar chart where some actual >numeric information can also be provided without >requiring tool tips. The units will also give a much better sense of quantity. >Comparing static bar charts over time is also significantly easier. Personally i think we are showing composition of DFS allocation, so pie chart is well suited in this case. Concrete numbers are already there in Summary and bar chart will show them numerically relative to each other. Also we don't have the dimension of time in jmx data. So, benefit of bar graph to show data changing over time is not applicable in this case. I am open to changing this if others feel that bar graph will serve the purpose better. was (Author: ajayydv): [~elgoiri],[~aw] thanks for your comments. >When talking about storage there are always two extra data points: time and >growth. Administrators are going to use this pie chart in >presentations. Then >use it again in 6 months. It ends being a horrible comparison because unless >the usage of the cluster hasn't >changed dramatically, it doesn't really >convey much information. Our UI's are stateless and are not suitable for comparing any data point over period of time. People usually rely on jmx data to do comparison over historical data. > Given that the tool tip is required really underscores this point. I have seen people working on hadoop from years but still getting confused b/w what DFS used and Non DFS used means in our summary table. Tooltip tries to address that. >Additionally, if I have more than one cluster, I'm going to pull up both NN >UIs and look at both charts simultaneously. Again, this doesn't >really convey >much information other than maybe the usage patterns are similar. It doesn't >convey an actual size. Yes, but this is current behavior(how our current UI is) and this change is not intended to address that. May be we can address it separately. >As an alternative, I think I'd much rather have a bar chart where some actual >numeric information can also be provided without >requiring tool tips. The >units will also give a much better sense of quantity. Comparing static bar >charts over time is also significantly >easier. Personally i think we are showing composition of DFS allocation, so pie chart is well suited in this case. Concrete numbers are already there in Summary and bar chart will show them numerically relative to each other. Also we don't have the dimension of time in jmx data. So, benefit of bar graph to show data changing over time is not applicable in this case. > Add pie chart in NN UI to show storage used > --- > > Key: HDFS-13059 > URL: https://issues.apache.org/jira/browse/HDFS-13059 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13059.001.patch, HDFS-13059.002.patch, > HDFS-13059.003.patch, Screen Shot 2018-01-24 at 1.58.28 PM.png, Screen Shot > 2018-01-24 at 1.58.33 PM.png, Screen Shot 2018-01-24 at 3.04.17 PM.png, > fed-capacity.png > > > This jira proposes to add a pie chart in NN UI to show storage used by: > * DFS Used (Tooltip : "Storage currently used for DFS.") > * DFS available (Tooltip : "Storage available for DFS use.") > * Non DFS Used (Tooltip : "Storage allocated for DFS but currently" + > " used by
[jira] [Updated] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDFS-13022:
Status: In Progress (was: Patch Available)
I move back this to the in progress state. I would like to test the final
version in a real kubernetes cluster again.
> Block Storage: Kubernetes dynamic persistent volume provisioner
> ---
>
> Key: HDFS-13022
> URL: https://issues.apache.org/jira/browse/HDFS-13022
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDFS-13022-HDFS-7240.001.patch,
> HDFS-13022-HDFS-7240.002.patch, HDFS-13022-HDFS-7240.003.patch
>
>
> {color:#FF}{color}
> With HDFS-13017 and HDFS-13018 the cblock/jscsi server could be used in a
> kubernetes cluster as the backend for iscsi persistent volumes.
> Unfortunatelly we need to create all the required cblocks manually with 'hdfs
> cblok -c user volume...' for all the Persistent Volumes.
>
> But it could be handled with a simple optional component. An additional
> service could listen on the kubernetes event stream. In case of new
> PersistentVolumeClaim (where the storageClassName is cblock) the cblock
> server could create cblock in advance AND create the persistent volume could
> be created.
>
> The code is very simple, and this additional component could be optional in
> the cblock server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13059) Add pie chart in NN UI to show storage used
[ https://issues.apache.org/jira/browse/HDFS-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339525#comment-16339525 ] Ajay Kumar commented on HDFS-13059: --- [~elgoiri],[~aw] thanks for your comments. >When talking about storage there are always two extra data points: time and >growth. Administrators are going to use this pie chart in >presentations. Then >use it again in 6 months. It ends being a horrible comparison because unless >the usage of the cluster hasn't >changed dramatically, it doesn't really >convey much information. Our UI's are stateless and are not suitable for comparing any data point over period of time. People usually rely on jmx data to do comparison over historical data. > Given that the tool tip is required really underscores this point. I have seen people working on hadoop from years but still getting confused b/w what DFS used and Non DFS used means in our summary table. Tooltip tries to address that. >Additionally, if I have more than one cluster, I'm going to pull up both NN >UIs and look at both charts simultaneously. Again, this doesn't >really convey >much information other than maybe the usage patterns are similar. It doesn't >convey an actual size. Yes, but this is current behavior(how our current UI is) and this change is not intended to address that. May be we can address it separately. >As an alternative, I think I'd much rather have a bar chart where some actual >numeric information can also be provided without >requiring tool tips. The >units will also give a much better sense of quantity. Comparing static bar >charts over time is also significantly >easier. Personally i think we are showing composition of DFS allocation, so pie chart is well suited in this case. Concrete numbers are already there in Summary and bar chart will show them numerically relative to each other. Also we don't have the dimension of time in jmx data. So, benefit of bar graph to show data changing over time is not applicable in this case. > Add pie chart in NN UI to show storage used > --- > > Key: HDFS-13059 > URL: https://issues.apache.org/jira/browse/HDFS-13059 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13059.001.patch, HDFS-13059.002.patch, > HDFS-13059.003.patch, Screen Shot 2018-01-24 at 1.58.28 PM.png, Screen Shot > 2018-01-24 at 1.58.33 PM.png, Screen Shot 2018-01-24 at 3.04.17 PM.png, > fed-capacity.png > > > This jira proposes to add a pie chart in NN UI to show storage used by: > * DFS Used (Tooltip : "Storage currently used for DFS.") > * DFS available (Tooltip : "Storage available for DFS use.") > * Non DFS Used (Tooltip : "Storage allocated for DFS but currently" + > " used by Non DFS storage.") > Tooltip will help users better understand what these terms mean. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13059) Add pie chart in NN UI to show storage used
[ https://issues.apache.org/jira/browse/HDFS-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339502#comment-16339502 ] Allen Wittenauer commented on HDFS-13059: - I guess I've failed to communicate my point here. :( When talking about storage there are always two extra data points: time and growth. Administrators are going to use this pie chart in presentations. Then use it again in 6 months. It ends being a horrible comparison because unless the usage of the cluster hasn't changed dramatically, it doesn't really convey much information. Given that the tool tip is required really underscores this point. Additionally, if I have more than one cluster, I'm going to pull up both NN UIs and look at both charts simultaneously. Again, this doesn't really convey much information other than maybe the usage patterns are similar. It doesn't convey an actual size. There's also a potential accessibility problem here, but we should probably consult with an expert. As an alternative, I think I'd much rather have a bar chart where some actual numeric information can also be provided without requiring tool tips. The units will also give a much better sense of quantity. Comparing static bar charts over time is also significantly easier. > Add pie chart in NN UI to show storage used > --- > > Key: HDFS-13059 > URL: https://issues.apache.org/jira/browse/HDFS-13059 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13059.001.patch, HDFS-13059.002.patch, > HDFS-13059.003.patch, Screen Shot 2018-01-24 at 1.58.28 PM.png, Screen Shot > 2018-01-24 at 1.58.33 PM.png, Screen Shot 2018-01-24 at 3.04.17 PM.png, > fed-capacity.png > > > This jira proposes to add a pie chart in NN UI to show storage used by: > * DFS Used (Tooltip : "Storage currently used for DFS.") > * DFS available (Tooltip : "Storage available for DFS use.") > * Non DFS Used (Tooltip : "Storage allocated for DFS but currently" + > " used by Non DFS storage.") > Tooltip will help users better understand what these terms mean. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13042) RBF: Heartbeat Router State
[ https://issues.apache.org/jira/browse/HDFS-13042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339500#comment-16339500 ] Íñigo Goiri commented on HDFS-13042: Thanks [~linyiqun] for the review and the commit. I'm moving into the subtasks. > RBF: Heartbeat Router State > --- > > Key: HDFS-13042 > URL: https://issues.apache.org/jira/browse/HDFS-13042 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Labels: RBF > Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1 > > Attachments: HDFS-13042.000.patch, HDFS-13042.001.patch, > HDFS-13042.002.patch, HDFS-13042.003.patch > > > The Router should heartbeat its state to the State Store. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13059) Add pie chart in NN UI to show storage used
[ https://issues.apache.org/jira/browse/HDFS-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13059: --- Attachment: (was: fed-capacity.png) > Add pie chart in NN UI to show storage used > --- > > Key: HDFS-13059 > URL: https://issues.apache.org/jira/browse/HDFS-13059 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13059.001.patch, HDFS-13059.002.patch, > HDFS-13059.003.patch, Screen Shot 2018-01-24 at 1.58.28 PM.png, Screen Shot > 2018-01-24 at 1.58.33 PM.png, Screen Shot 2018-01-24 at 3.04.17 PM.png, > fed-capacity.png > > > This jira proposes to add a pie chart in NN UI to show storage used by: > * DFS Used (Tooltip : "Storage currently used for DFS.") > * DFS available (Tooltip : "Storage available for DFS use.") > * Non DFS Used (Tooltip : "Storage allocated for DFS but currently" + > " used by Non DFS storage.") > Tooltip will help users better understand what these terms mean. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13059) Add pie chart in NN UI to show storage used
[ https://issues.apache.org/jira/browse/HDFS-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13059: --- Attachment: fed-capacity.png > Add pie chart in NN UI to show storage used > --- > > Key: HDFS-13059 > URL: https://issues.apache.org/jira/browse/HDFS-13059 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13059.001.patch, HDFS-13059.002.patch, > HDFS-13059.003.patch, Screen Shot 2018-01-24 at 1.58.28 PM.png, Screen Shot > 2018-01-24 at 1.58.33 PM.png, Screen Shot 2018-01-24 at 3.04.17 PM.png, > fed-capacity.png > > > This jira proposes to add a pie chart in NN UI to show storage used by: > * DFS Used (Tooltip : "Storage currently used for DFS.") > * DFS available (Tooltip : "Storage available for DFS use.") > * Non DFS Used (Tooltip : "Storage allocated for DFS but currently" + > " used by Non DFS storage.") > Tooltip will help users better understand what these terms mean. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13059) Add pie chart in NN UI to show storage used
[ https://issues.apache.org/jira/browse/HDFS-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339494#comment-16339494 ] Íñigo Goiri commented on HDFS-13059: To give a little bit more context, one of the reasons why I asked for the refactor of the visualization functions is because I'd like to use them in Router-based federation. Actually, this goes into the point that [~aw] asked about how to compare multiple clusters. Currently, in RBF we have the following: !fed-capacity.png! This is a good starting point but there is no way to compare the sizes across subclusters. I'd like to use the pie chart or the histogram to show the space across subclusters too. > Add pie chart in NN UI to show storage used > --- > > Key: HDFS-13059 > URL: https://issues.apache.org/jira/browse/HDFS-13059 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13059.001.patch, HDFS-13059.002.patch, > HDFS-13059.003.patch, Screen Shot 2018-01-24 at 1.58.28 PM.png, Screen Shot > 2018-01-24 at 1.58.33 PM.png, Screen Shot 2018-01-24 at 3.04.17 PM.png, > fed-capacity.png > > > This jira proposes to add a pie chart in NN UI to show storage used by: > * DFS Used (Tooltip : "Storage currently used for DFS.") > * DFS available (Tooltip : "Storage available for DFS use.") > * Non DFS Used (Tooltip : "Storage allocated for DFS but currently" + > " used by Non DFS storage.") > Tooltip will help users better understand what these terms mean. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13059) Add pie chart in NN UI to show storage used
[ https://issues.apache.org/jira/browse/HDFS-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13059: --- Attachment: fed-capacity.png > Add pie chart in NN UI to show storage used > --- > > Key: HDFS-13059 > URL: https://issues.apache.org/jira/browse/HDFS-13059 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13059.001.patch, HDFS-13059.002.patch, > HDFS-13059.003.patch, Screen Shot 2018-01-24 at 1.58.28 PM.png, Screen Shot > 2018-01-24 at 1.58.33 PM.png, Screen Shot 2018-01-24 at 3.04.17 PM.png, > fed-capacity.png > > > This jira proposes to add a pie chart in NN UI to show storage used by: > * DFS Used (Tooltip : "Storage currently used for DFS.") > * DFS available (Tooltip : "Storage available for DFS use.") > * Non DFS Used (Tooltip : "Storage allocated for DFS but currently" + > " used by Non DFS storage.") > Tooltip will help users better understand what these terms mean. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339479#comment-16339479
]
Elek, Marton commented on HDFS-13022:
-
Thank you very much the review, [~msingh]. You are all right. I addressed all
the comments (license, javadoc, init, testname,...)
Also fixed a few typo like Cblock -> CBlock and tested and fixed the default
configuration values (TestOzoneConfigurationFields).
New version is uploaded.
> Block Storage: Kubernetes dynamic persistent volume provisioner
> ---
>
> Key: HDFS-13022
> URL: https://issues.apache.org/jira/browse/HDFS-13022
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDFS-13022-HDFS-7240.001.patch,
> HDFS-13022-HDFS-7240.002.patch, HDFS-13022-HDFS-7240.003.patch
>
>
> {color:#FF}{color}
> With HDFS-13017 and HDFS-13018 the cblock/jscsi server could be used in a
> kubernetes cluster as the backend for iscsi persistent volumes.
> Unfortunatelly we need to create all the required cblocks manually with 'hdfs
> cblok -c user volume...' for all the Persistent Volumes.
>
> But it could be handled with a simple optional component. An additional
> service could listen on the kubernetes event stream. In case of new
> PersistentVolumeClaim (where the storageClassName is cblock) the cblock
> server could create cblock in advance AND create the persistent volume could
> be created.
>
> The code is very simple, and this additional component could be optional in
> the cblock server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDFS-13022:
Attachment: HDFS-13022-HDFS-7240.003.patch
> Block Storage: Kubernetes dynamic persistent volume provisioner
> ---
>
> Key: HDFS-13022
> URL: https://issues.apache.org/jira/browse/HDFS-13022
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDFS-13022-HDFS-7240.001.patch,
> HDFS-13022-HDFS-7240.002.patch, HDFS-13022-HDFS-7240.003.patch
>
>
> {color:#FF}{color}
> With HDFS-13017 and HDFS-13018 the cblock/jscsi server could be used in a
> kubernetes cluster as the backend for iscsi persistent volumes.
> Unfortunatelly we need to create all the required cblocks manually with 'hdfs
> cblok -c user volume...' for all the Persistent Volumes.
>
> But it could be handled with a simple optional component. An additional
> service could listen on the kubernetes event stream. In case of new
> PersistentVolumeClaim (where the storageClassName is cblock) the cblock
> server could create cblock in advance AND create the persistent volume could
> be created.
>
> The code is very simple, and this additional component could be optional in
> the cblock server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13059) Add pie chart in NN UI to show storage used
[ https://issues.apache.org/jira/browse/HDFS-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13059: -- Attachment: HDFS-13059.003.patch > Add pie chart in NN UI to show storage used > --- > > Key: HDFS-13059 > URL: https://issues.apache.org/jira/browse/HDFS-13059 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13059.001.patch, HDFS-13059.002.patch, > HDFS-13059.003.patch, Screen Shot 2018-01-24 at 1.58.28 PM.png, Screen Shot > 2018-01-24 at 1.58.33 PM.png, Screen Shot 2018-01-24 at 3.04.17 PM.png > > > This jira proposes to add a pie chart in NN UI to show storage used by: > * DFS Used (Tooltip : "Storage currently used for DFS.") > * DFS available (Tooltip : "Storage available for DFS use.") > * Non DFS Used (Tooltip : "Storage allocated for DFS but currently" + > " used by Non DFS storage.") > Tooltip will help users better understand what these terms mean. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13059) Add pie chart in NN UI to show storage used
[
https://issues.apache.org/jira/browse/HDFS-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339471#comment-16339471
]
Ajay Kumar commented on HDFS-13059:
---
[~elgoiri], Created [HDFS-13066] to refactor bar graph related code in new js.
Addressed rest of the comments in patch v3.
[~aw] Article you shared compared pie chart with other forms of visualizations.
Writer is more critical about pie chart not conveying some additional
information (number/figures) specially when data points in hands are much more.
This is not always required. Sometimes we just want to convey relative value of
datapoints in hand. As [~elgoiri] pointed out article does see use of pie chart
useful when comparing small set of datapoints. To be more specific pie chart
are simple enough to show composition of 2-3 data points.Here we simply want to
show current composition of storage reserved for DFS. (3 data points) Tooltip
will show exact % along with its brief summary.
Writer in that article compares pie chart with other forms of visualization. In
our case we have to compare it with what we have right now. i.e
text/numeric-table, which sounds like an improvement.
a) How does one compare the pie chart from multiple clusters?
Current dust js code iterate over array of distinct nn jmx data. Although i
have not tested it for multiple cluster, this change will have same behavior as
existing code.
b) Given that there will be times when two or more of the slices of the pie
charts will be nearly the same size, what does that tell the user?
Tooltip will show exact %. Users don't have to worry until DFS free becomes too
low. Also please keep in mind this change doesn't remove any of the existing
data we already show. So if someone was referring to {{Summary}} table to see
exact values they can still do so.
c) When storage is added to a cluster, how is that reflected in the pie chart
over time?
Pie chart or even "Summary" table is rendered from jmx data. It will update as
soon as jmx data is updated.
> Add pie chart in NN UI to show storage used
> ---
>
> Key: HDFS-13059
> URL: https://issues.apache.org/jira/browse/HDFS-13059
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-13059.001.patch, HDFS-13059.002.patch,
> HDFS-13059.003.patch, Screen Shot 2018-01-24 at 1.58.28 PM.png, Screen Shot
> 2018-01-24 at 1.58.33 PM.png, Screen Shot 2018-01-24 at 3.04.17 PM.png
>
>
> This jira proposes to add a pie chart in NN UI to show storage used by:
> * DFS Used (Tooltip : "Storage currently used for DFS.")
> * DFS available (Tooltip : "Storage available for DFS use.")
> * Non DFS Used (Tooltip : "Storage allocated for DFS but currently" +
> " used by Non DFS storage.")
> Tooltip will help users better understand what these terms mean.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13066) Refactor barchart in dfshealth.js to separate js file
Ajay Kumar created HDFS-13066:
-
Summary: Refactor barchart in dfshealth.js to separate js file
Key: HDFS-13066
URL: https://issues.apache.org/jira/browse/HDFS-13066
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ajay Kumar
Assignee: Ajay Kumar
Refactor barchart in {{dfshealth.js}} to separate js file to make visualization
code reusable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13065) TestErasureCodingMultipleRacks#testSkewedRack3 is failing
[ https://issues.apache.org/jira/browse/HDFS-13065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HDFS-13065: -- Attachment: HDFS-13065.002.patch > TestErasureCodingMultipleRacks#testSkewedRack3 is failing > - > > Key: HDFS-13065 > URL: https://issues.apache.org/jira/browse/HDFS-13065 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Fix For: 3.0.1 > > Attachments: HDFS-13065.001.patch, HDFS-13065.002.patch > > > Test fails intermittently. > java.lang.AssertionError: expected:<9> but was:<7> > at > org.apache.hadoop.hdfs.TestErasureCodingMultipleRacks.testSkewedRack3(TestErasureCodingMultipleRacks.java:178) > 2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] DEBUG > net.NetworkTopology > (DFSNetworkTopology.java:chooseRandomWithStorageTypeTwoTrial(119)) - No node > to choose. > 2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] WARN > blockmanagement.BlockPlacementPolicy > (BlockPlacementPolicyRackFaultTolerant.java:chooseTargetInOrder(142)) - Only > able to place 7 of total expected 9 (maxNodesPerRack=2, numOfReplicas=4) > nodes evenly across racks, falling back to evenly place on the remaining > racks. This may not guarantee rack-level fault tolerance. Please check if the > racks are configured properly. > 2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] DEBUG > blockmanagement.BlockPlacementPolicy > (BlockPlacementPolicyRackFaultTolerant.java:chooseTargetInOrder(148)) - > Caught exception was: > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException: > > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:792) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant.chooseOnce(BlockPlacementPolicyRackFaultTolerant.java:224) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant.chooseTargetInOrder(BlockPlacementPolicyRackFaultTolerant.java:139) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:392) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:268) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:121) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:137) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2091) > at > org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:287) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2602) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:864) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:549) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13065) TestErasureCodingMultipleRacks#testSkewedRack3 is failing
[ https://issues.apache.org/jira/browse/HDFS-13065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HDFS-13065: -- Status: Patch Available (was: Open) Fixed checkstyle error. Other errors were unrelated. > TestErasureCodingMultipleRacks#testSkewedRack3 is failing > - > > Key: HDFS-13065 > URL: https://issues.apache.org/jira/browse/HDFS-13065 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Fix For: 3.0.1 > > Attachments: HDFS-13065.001.patch, HDFS-13065.002.patch > > > Test fails intermittently. > java.lang.AssertionError: expected:<9> but was:<7> > at > org.apache.hadoop.hdfs.TestErasureCodingMultipleRacks.testSkewedRack3(TestErasureCodingMultipleRacks.java:178) > 2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] DEBUG > net.NetworkTopology > (DFSNetworkTopology.java:chooseRandomWithStorageTypeTwoTrial(119)) - No node > to choose. > 2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] WARN > blockmanagement.BlockPlacementPolicy > (BlockPlacementPolicyRackFaultTolerant.java:chooseTargetInOrder(142)) - Only > able to place 7 of total expected 9 (maxNodesPerRack=2, numOfReplicas=4) > nodes evenly across racks, falling back to evenly place on the remaining > racks. This may not guarantee rack-level fault tolerance. Please check if the > racks are configured properly. > 2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] DEBUG > blockmanagement.BlockPlacementPolicy > (BlockPlacementPolicyRackFaultTolerant.java:chooseTargetInOrder(148)) - > Caught exception was: > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException: > > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:792) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant.chooseOnce(BlockPlacementPolicyRackFaultTolerant.java:224) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant.chooseTargetInOrder(BlockPlacementPolicyRackFaultTolerant.java:139) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:392) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:268) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:121) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:137) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2091) > at > org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:287) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2602) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:864) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:549) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13065) TestErasureCodingMultipleRacks#testSkewedRack3 is failing
[ https://issues.apache.org/jira/browse/HDFS-13065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HDFS-13065: -- Status: Open (was: Patch Available) > TestErasureCodingMultipleRacks#testSkewedRack3 is failing > - > > Key: HDFS-13065 > URL: https://issues.apache.org/jira/browse/HDFS-13065 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Fix For: 3.0.1 > > Attachments: HDFS-13065.001.patch, HDFS-13065.002.patch > > > Test fails intermittently. > java.lang.AssertionError: expected:<9> but was:<7> > at > org.apache.hadoop.hdfs.TestErasureCodingMultipleRacks.testSkewedRack3(TestErasureCodingMultipleRacks.java:178) > 2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] DEBUG > net.NetworkTopology > (DFSNetworkTopology.java:chooseRandomWithStorageTypeTwoTrial(119)) - No node > to choose. > 2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] WARN > blockmanagement.BlockPlacementPolicy > (BlockPlacementPolicyRackFaultTolerant.java:chooseTargetInOrder(142)) - Only > able to place 7 of total expected 9 (maxNodesPerRack=2, numOfReplicas=4) > nodes evenly across racks, falling back to evenly place on the remaining > racks. This may not guarantee rack-level fault tolerance. Please check if the > racks are configured properly. > 2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] DEBUG > blockmanagement.BlockPlacementPolicy > (BlockPlacementPolicyRackFaultTolerant.java:chooseTargetInOrder(148)) - > Caught exception was: > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException: > > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:792) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant.chooseOnce(BlockPlacementPolicyRackFaultTolerant.java:224) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant.chooseTargetInOrder(BlockPlacementPolicyRackFaultTolerant.java:139) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:392) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:268) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:121) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:137) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2091) > at > org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:287) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2602) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:864) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:549) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13065) TestErasureCodingMultipleRacks#testSkewedRack3 is failing
[
https://issues.apache.org/jira/browse/HDFS-13065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339213#comment-16339213
]
genericqa commented on HDFS-13065:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 58s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}161m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSClientExcludedNodes |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13065 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907660/HDFS-13065.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux f32afbd2459c 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 82cc6f6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/22810/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22810/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Comment Edited] (HDFS-13044) RBF: Add a safe mode for the Router
[
https://issues.apache.org/jira/browse/HDFS-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339120#comment-16339120
]
Yiqun Lin edited comment on HDFS-13044 at 1/25/18 12:00 PM:
Hi [~elgoiri], there are some initial review comments from me.
*DFSConfigKeys.java*
1. Can we rename config name of {{DFS_ROUTER_SAFEMODE_CACHE_EXPIRATION_MS}}
from {{dfs.federation.router.safemode.cacheexpiration.count}} to
{{dfs.federation.router.safemode.cache-expiration.interval}}? Why not use {{2
or 3 * DFS_ROUTER_CACHE_TIME_TO_LIVE_MS_DEFAULT}} as the default expiration
interval time? Looks these two config are related, this will be better than
just assigning a new value.
2. It would be better to make these configs support multiple time units
suffixes.
3. Please document these configs in {{hdfs-default.xml}}.
*Router.java*
1.Following lines need to update.
{code:java}
+if (this.safemodeService == null) {
+ // Router is running now
+ // TODO
+}
{code}
2. line125. This comment seems to be {{/** The start time of the Router. */}}.
This was something I am missing caught before.
*RouterRpcServer.java*
line425: Why not define and throw a sub-exception of {{SafeModeException}}?
line426: Missing "is" before {{in safe mode and cannot handle}}.
line446: Can we rename {{isSafeMode}} to {{isInSafeMode}}?
*RouterSafemodeService.java*
line33 and 36: Seems {{FederationStateStoreService}} is non-exist.
line55: Maybe we need to define a new field named {{enterTime}} to calculate
elapsed time during safemoe. And {{enterTime}} will be updated in {{enter}}
method every time. We may experience enter/leave safemode many times in real
env.
line82: {{synchronized}} is not needed since {{MutableGaugeInt#set}} is
thread-safe to use. It uses AtomicInteger type to record value inside.
line84: Can we print the elapsed time during safemode as well?
line90: {{(int)}} is not needed, it already did in
{{RouterMetrics#setSafeModeTime}}.
line110 and 117: Can we print the interval time with {{MILLISECONDS}} format?
There is a chance the value loosing accurate. For example, we set the interval
time 999ms, it will return 0 when convert to second format.
*TestRouterSafemode.java*
line43: The javadoc looks confused. There are two duplicate {{the}}.
{{SafeModeTimer}} is non-exist.
line116: We are just for the testing, no need to sleep so long time. We can
greatly decrease the interval time tested in this class. The same problem in
{{testRouterEnterSafemode}}.
Additional two test cases we may needed:
1. Specific test for {{dfs.federation.router.safemode.startup.interval}}. That
is say, router won't leaf safemode until over interval time.
2. The case that RouterRpcServer rejects the reuquests when it's in safemode.
{quote}We could also add an API in the Router Admin to set the safe mode by
hand.
{quote}
+1 for this proposal. We can file new JIRA for the tracking.
In addition, we need to document the meaning of router states. At least I see
the semantic of safemode in RBF is different with that in normal HDFS.
was (Author: linyiqun):
Hi [~elgoiri], there are some initial review comments from me.
*DFSConfigKeys.java*
1. Can we rename config name of {{DFS_ROUTER_SAFEMODE_CACHE_EXPIRATION_MS}}
from {{dfs.federation.router.safemode.cacheexpiration.count}} to
{{dfs.federation.router.safemode.cache-expiration.interval}}? Why not use {{2
or 3 * DFS_ROUTER_CACHE_TIME_TO_LIVE_MS_DEFAULT}} as the default expiration
interval time? Looks these two config are related, this will be better than
just assigning a new value.
2. It would be better to make these configs support multiple time units
suffixes.
3. Please document these configs in {{hdfs-default.xml}}.
*Router.java*
1.Following lines need to update.
{code:java}
+if (this.safemodeService == null) {
+ // Router is running now
+ // TODO
+}
{code}
2. line125. This comment seems to be {{/** The start time of the Router. */}}.
This was something I am missing caught before.
*RouterRpcServer.java*
line425: Why not define and throw a sub-exception of {{SafeModeException}}?
line426: Missing "is" before {{in safe mode and cannot handle}}.
line446: Can we rename {{isSafeMode}} to {{isInSafeMode}}?
*RouterSafemodeService.java*
line33 and 36: Seems {{FederationStateStoreService}} is non-exist.
line55: Maybe we need to define a new field named {{enterTime}} to calculate
elapsed time during safemoe. And {{enterTime}} will be updated in {{enter}}
method every time. We may experience enter/leave safemode many times in real
env.
line82: {{synchronized}} is not needed since {{MutableGaugeInt#set}} is
thread-safe to use. It uses AtomicInteger type to record value inside.
line84: Can we print the elapsed time during safemode as well?
line90: {{(int)}} is not needed, it already did in
{{RouterMetrics#setSafeModeTime}}.
line110 and
