[jira] [Commented] (HDFS-13639) SlotReleaser is not fast enough
[
https://issues.apache.org/jira/browse/HDFS-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307586#comment-17307586
]
Lisheng Sun commented on HDFS-13639:
Hi, [~sodonnell]
This patch is running in production for a long time.
The concurrency of short-circuit read has been improved greatly in the Hbase
sence.
> SlotReleaser is not fast enough
> ---
>
> Key: HDFS-13639
> URL: https://issues.apache.org/jira/browse/HDFS-13639
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.4.0, 2.6.0, 3.0.2
> Environment: 1. YCSB:
> {color:#00} recordcount=20
> fieldcount=1
> fieldlength=1000
> operationcount=1000
>
> workload=com.yahoo.ycsb.workloads.CoreWorkload
>
> table=ycsb-test
> columnfamily=C
> readproportion=1
> updateproportion=0
> insertproportion=0
> scanproportion=0
>
> maxscanlength=0
> requestdistribution=zipfian
>
> # default
> readallfields=true
> writeallfields=true
> scanlengthdistribution=constan{color}
> {color:#00}2. datanode:{color}
> -Xmx2048m -Xms2048m -Xmn1024m -XX:MaxDirectMemorySize=1024m
> -XX:MaxPermSize=256m -Xloggc:$run_dir/stdout/datanode_gc_${start_time}.log
> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=$log_dir -XX:+PrintGCApplicationStoppedTime
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSParallelRemarkEnabled
> -XX:+CMSClassUnloadingEnabled -XX:CMSMaxAbortablePrecleanTime=1
> -XX:+CMSScavengeBeforeRemark -XX:+PrintPromotionFailure
> -XX:+CMSConcurrentMTEnabled -XX:+ExplicitGCInvokesConcurrent
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -verbose:gc -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps
> {color:#00}3. regionserver:{color}
> {color:#00}-Xmx10g -Xms10g -XX:MaxDirectMemorySize=10g
> -XX:MaxGCPauseMillis=150 -XX:MaxTenuringThreshold=2
> -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=5
> -Xloggc:$run_dir/stdout/regionserver_gc_${start_time}.log -Xss256k
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$log_dir -verbose:gc
> -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy
> -XX:+PrintTenuringDistribution -XX:+PrintSafepointStatistics
> -XX:PrintSafepointStatisticsCount=1 -XX:PrintFLSStatistics=1
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=128m
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=65
> -XX:+ParallelRefProcEnabled -XX:ConcGCThreads=4 -XX:ParallelGCThreads=16
> -XX:G1HeapRegionSize=32m -XX:G1MixedGCCountTarget=64
> -XX:G1OldCSetRegionThresholdPercent=5{color}
> {color:#00}block cache is disabled:{color}{color:#00}
> hbase.bucketcache.size
> 0.9
> {color}
>
>Reporter: Gang Xie
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: HDFS-13639-2.4.diff, HDFS-13639.001.patch,
> HDFS-13639.002.patch, ShortCircuitCache_new_slotReleaser.diff,
> perf_after_improve_SlotReleaser.png, perf_before_improve_SlotReleaser.png
>
>
> When test the performance of the ShortCircuit Read of the HDFS with YCSB, we
> find that SlotReleaser of the ShortCircuitCache has some performance issue.
> The problem is that, the qps of the slot releasing could only reach to 1000+
> while the qps of the slot allocating is ~3000. This means that the replica
> info on datanode could not be released in time, which causes a lot of GCs and
> finally full GCs.
>
> The fireflame graph shows that SlotReleaser spends a lot of time to do domain
> socket connecting and throw/catching the exception when close the domain
> socket and its streams. It doesn't make any sense to do the connecting and
> closing each time. Each time when we connect to the domain socket, Datanode
> allocates a new thread to free the slot. There are a lot of initializing
> work, and it's costly. We need reuse the domain socket.
>
> After switch to reuse the domain socket(see diff attached), we get great
> improvement(see the perf):
> # without reusing the domain socket, the get qps of the YCSB getting worse
> and worse, and after about 45 mins, full GC starts. When we reuse the domain
> socket, no full GC found, and the stress test could be finished smoothly, the
> qps of allocating and releasing match.
> # Due to the datanode young GC, without the improvement, the YCSB get qps is
> even smaller than the one with the improvement, ~3700 VS ~4200.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe,
[jira] [Commented] (HDFS-15787) Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
[
https://issues.apache.org/jira/browse/HDFS-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305617#comment-17305617
]
Lisheng Sun commented on HDFS-15787:
Hi [~ayushtkn]，Could we commit this patch to trunk if no one comments this
issue?
> Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
>
>
> Key: HDFS-15787
> URL: https://issues.apache.org/jira/browse/HDFS-15787
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-15787.001.patch, HDFS-15787.002.patch
>
>
> The method of FSNamesystem#internalReleaseLease() as follow:
>
> {code:java}
> boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
> String recoveryLeaseHolder) throws IOException {
> ...
> // Start recovery of the last block for this file
> // Only do so if there is no ongoing recovery for this block,
> // or the previous recovery for this block timed out.
> if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
> long blockRecoveryId = nextGenerationStamp(
> blockManager.isLegacyBlock(lastBlock));
> if(copyOnTruncate) {
> lastBlock.setGenerationStamp(blockRecoveryId);
> } else if(truncateRecovery) {
> recoveryBlock.setGenerationStamp(blockRecoveryId);
> }
> uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
> // Cannot close file right now, since the last block requires recovery.
> // This may potentially cause infinite loop in lease recovery
> // if there are no valid replicas on data-nodes.
> NameNode.stateChangeLog.warn(
> "DIR* NameSystem.internalReleaseLease: " +
> "File " + src + " has not been closed." +
> " Lease recovery is in progress. " +
> "RecoveryId = " + blockRecoveryId + " for block " + lastBlock);
> }
> lease = reassignLease(lease, src, recoveryLeaseHolder, pendingFile);
> leaseManager.renewLease(lease);
> break;
> }
> return false;
> }
> {code}
> Call LeaseManager#renewLease in
> FSNamesystem#reassignLease=>FSNamesystem#reassignLeaseInternal.
> So no need to call LeaseManager#renewLease again after
> leaseManager#renewLease.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15809) DeadNodeDetector doesn't remove live nodes from dead node set.
[ https://issues.apache.org/jira/browse/HDFS-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-15809: --- Fix Version/s: 3.4.0 > DeadNodeDetector doesn't remove live nodes from dead node set. > -- > > Key: HDFS-15809 > URL: https://issues.apache.org/jira/browse/HDFS-15809 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15809.001.patch, HDFS-15809.002.patch, > HDFS-15809.003.patch, HDFS-15809.004.patch, HDFS-15809.005.patch, > HDFS-15809.006.patch, HDFS-15809.007.patch > > > We found the dead node detector might never remove the alive nodes from the > dead node set in a big cluster. For example: > # 200 nodes are added to the dead node set by DeadNodeDetector. > # DeadNodeDetector#checkDeadNodes() adds 100 nodes to the > deadNodesProbeQueue because the queue limited length is 100. > # The probe threads start working and probe 30 nodes. > # DeadNodeDetector#checkDeadNodes() is scheduled again. It iterates the dead > node set and adds 30 nodes to the deadNodesProbeQueue. But the order is the > same as the last time. So the 30 nodes that has already been probed are added > to the queue again. > # Repeat 3 and 4. But we always add the first 30 nodes from the dead set. If > they are all dead then the live nodes behind them could never be recovered. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15809) DeadNodeDetector doesn't remove live nodes from dead node set.
[ https://issues.apache.org/jira/browse/HDFS-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301363#comment-17301363 ] Lisheng Sun commented on HDFS-15809: Committed to trunk. Thanx [~LiJinglun] for the contribution. > DeadNodeDetector doesn't remove live nodes from dead node set. > -- > > Key: HDFS-15809 > URL: https://issues.apache.org/jira/browse/HDFS-15809 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15809.001.patch, HDFS-15809.002.patch, > HDFS-15809.003.patch, HDFS-15809.004.patch, HDFS-15809.005.patch, > HDFS-15809.006.patch, HDFS-15809.007.patch > > > We found the dead node detector might never remove the alive nodes from the > dead node set in a big cluster. For example: > # 200 nodes are added to the dead node set by DeadNodeDetector. > # DeadNodeDetector#checkDeadNodes() adds 100 nodes to the > deadNodesProbeQueue because the queue limited length is 100. > # The probe threads start working and probe 30 nodes. > # DeadNodeDetector#checkDeadNodes() is scheduled again. It iterates the dead > node set and adds 30 nodes to the deadNodesProbeQueue. But the order is the > same as the last time. So the 30 nodes that has already been probed are added > to the queue again. > # Repeat 3 and 4. But we always add the first 30 nodes from the dead set. If > they are all dead then the live nodes behind them could never be recovered. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15809) DeadNodeDetector doesn't remove live nodes from dead node set.
[ https://issues.apache.org/jira/browse/HDFS-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-15809: --- Resolution: Fixed Status: Resolved (was: Patch Available) > DeadNodeDetector doesn't remove live nodes from dead node set. > -- > > Key: HDFS-15809 > URL: https://issues.apache.org/jira/browse/HDFS-15809 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15809.001.patch, HDFS-15809.002.patch, > HDFS-15809.003.patch, HDFS-15809.004.patch, HDFS-15809.005.patch, > HDFS-15809.006.patch, HDFS-15809.007.patch > > > We found the dead node detector might never remove the alive nodes from the > dead node set in a big cluster. For example: > # 200 nodes are added to the dead node set by DeadNodeDetector. > # DeadNodeDetector#checkDeadNodes() adds 100 nodes to the > deadNodesProbeQueue because the queue limited length is 100. > # The probe threads start working and probe 30 nodes. > # DeadNodeDetector#checkDeadNodes() is scheduled again. It iterates the dead > node set and adds 30 nodes to the deadNodesProbeQueue. But the order is the > same as the last time. So the 30 nodes that has already been probed are added > to the queue again. > # Repeat 3 and 4. But we always add the first 30 nodes from the dead set. If > they are all dead then the live nodes behind them could never be recovered. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15809) DeadNodeDetector doesn't remove live nodes from dead node set.
[ https://issues.apache.org/jira/browse/HDFS-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297395#comment-17297395 ] Lisheng Sun edited comment on HDFS-15809 at 3/8/21, 2:15 PM: - Thank [~LiJinglun] for your patience work. LGFM . +1 for [^HDFS-15809.007.patch] . was (Author: leosun08): Thank [~LiJinglun] for your patience work. LGFM . +1 for https://issues.apache.org/jira/secure/attachment/13021765/HDFS-15809.007.patch > DeadNodeDetector doesn't remove live nodes from dead node set. > -- > > Key: HDFS-15809 > URL: https://issues.apache.org/jira/browse/HDFS-15809 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15809.001.patch, HDFS-15809.002.patch, > HDFS-15809.003.patch, HDFS-15809.004.patch, HDFS-15809.005.patch, > HDFS-15809.006.patch, HDFS-15809.007.patch > > > We found the dead node detector might never remove the alive nodes from the > dead node set in a big cluster. For example: > # 200 nodes are added to the dead node set by DeadNodeDetector. > # DeadNodeDetector#checkDeadNodes() adds 100 nodes to the > deadNodesProbeQueue because the queue limited length is 100. > # The probe threads start working and probe 30 nodes. > # DeadNodeDetector#checkDeadNodes() is scheduled again. It iterates the dead > node set and adds 30 nodes to the deadNodesProbeQueue. But the order is the > same as the last time. So the 30 nodes that has already been probed are added > to the queue again. > # Repeat 3 and 4. But we always add the first 30 nodes from the dead set. If > they are all dead then the live nodes behind them could never be recovered. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15809) DeadNodeDetector doesn't remove live nodes from dead node set.
[ https://issues.apache.org/jira/browse/HDFS-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297395#comment-17297395 ] Lisheng Sun commented on HDFS-15809: Thank [~LiJinglun] for your patience work. LGFM . +1 for https://issues.apache.org/jira/secure/attachment/13021765/HDFS-15809.007.patch > DeadNodeDetector doesn't remove live nodes from dead node set. > -- > > Key: HDFS-15809 > URL: https://issues.apache.org/jira/browse/HDFS-15809 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15809.001.patch, HDFS-15809.002.patch, > HDFS-15809.003.patch, HDFS-15809.004.patch, HDFS-15809.005.patch, > HDFS-15809.006.patch, HDFS-15809.007.patch > > > We found the dead node detector might never remove the alive nodes from the > dead node set in a big cluster. For example: > # 200 nodes are added to the dead node set by DeadNodeDetector. > # DeadNodeDetector#checkDeadNodes() adds 100 nodes to the > deadNodesProbeQueue because the queue limited length is 100. > # The probe threads start working and probe 30 nodes. > # DeadNodeDetector#checkDeadNodes() is scheduled again. It iterates the dead > node set and adds 30 nodes to the deadNodesProbeQueue. But the order is the > same as the last time. So the 30 nodes that has already been probed are added > to the queue again. > # Repeat 3 and 4. But we always add the first 30 nodes from the dead set. If > they are all dead then the live nodes behind them could never be recovered. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15787) Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
[
https://issues.apache.org/jira/browse/HDFS-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296486#comment-17296486
]
Lisheng Sun edited comment on HDFS-15787 at 3/6/21, 9:38 AM:
-
Hi [~brahmareddy] [~cdouglas] Could you have time to confirm this issue that
was involved in HDFS-11576 ? Thank you.
was (Author: leosun08):
Hi [~brahmareddy] [~cdouglas] Could you have time to confirm this issue that
was involved in HDFS-11576
> Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
>
>
> Key: HDFS-15787
> URL: https://issues.apache.org/jira/browse/HDFS-15787
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-15787.001.patch, HDFS-15787.002.patch
>
>
> The method of FSNamesystem#internalReleaseLease() as follow:
>
> {code:java}
> boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
> String recoveryLeaseHolder) throws IOException {
> ...
> // Start recovery of the last block for this file
> // Only do so if there is no ongoing recovery for this block,
> // or the previous recovery for this block timed out.
> if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
> long blockRecoveryId = nextGenerationStamp(
> blockManager.isLegacyBlock(lastBlock));
> if(copyOnTruncate) {
> lastBlock.setGenerationStamp(blockRecoveryId);
> } else if(truncateRecovery) {
> recoveryBlock.setGenerationStamp(blockRecoveryId);
> }
> uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
> // Cannot close file right now, since the last block requires recovery.
> // This may potentially cause infinite loop in lease recovery
> // if there are no valid replicas on data-nodes.
> NameNode.stateChangeLog.warn(
> "DIR* NameSystem.internalReleaseLease: " +
> "File " + src + " has not been closed." +
> " Lease recovery is in progress. " +
> "RecoveryId = " + blockRecoveryId + " for block " + lastBlock);
> }
> lease = reassignLease(lease, src, recoveryLeaseHolder, pendingFile);
> leaseManager.renewLease(lease);
> break;
> }
> return false;
> }
> {code}
> Call LeaseManager#renewLease in
> FSNamesystem#reassignLease=>FSNamesystem#reassignLeaseInternal.
> So no need to call LeaseManager#renewLease again after
> leaseManager#renewLease.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15787) Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
[
https://issues.apache.org/jira/browse/HDFS-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296486#comment-17296486
]
Lisheng Sun edited comment on HDFS-15787 at 3/6/21, 9:37 AM:
-
Hi [~brahmareddy] [~cdouglas] Could you have time to confirm this issue that
was involved in HDFS-11576
was (Author: leosun08):
Hi [~brahmareddy] [~cdouglas] Could you have time to confirm this
issue?[|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=cdouglas]
> Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
>
>
> Key: HDFS-15787
> URL: https://issues.apache.org/jira/browse/HDFS-15787
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-15787.001.patch, HDFS-15787.002.patch
>
>
> The method of FSNamesystem#internalReleaseLease() as follow:
>
> {code:java}
> boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
> String recoveryLeaseHolder) throws IOException {
> ...
> // Start recovery of the last block for this file
> // Only do so if there is no ongoing recovery for this block,
> // or the previous recovery for this block timed out.
> if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
> long blockRecoveryId = nextGenerationStamp(
> blockManager.isLegacyBlock(lastBlock));
> if(copyOnTruncate) {
> lastBlock.setGenerationStamp(blockRecoveryId);
> } else if(truncateRecovery) {
> recoveryBlock.setGenerationStamp(blockRecoveryId);
> }
> uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
> // Cannot close file right now, since the last block requires recovery.
> // This may potentially cause infinite loop in lease recovery
> // if there are no valid replicas on data-nodes.
> NameNode.stateChangeLog.warn(
> "DIR* NameSystem.internalReleaseLease: " +
> "File " + src + " has not been closed." +
> " Lease recovery is in progress. " +
> "RecoveryId = " + blockRecoveryId + " for block " + lastBlock);
> }
> lease = reassignLease(lease, src, recoveryLeaseHolder, pendingFile);
> leaseManager.renewLease(lease);
> break;
> }
> return false;
> }
> {code}
> Call LeaseManager#renewLease in
> FSNamesystem#reassignLease=>FSNamesystem#reassignLeaseInternal.
> So no need to call LeaseManager#renewLease again after
> leaseManager#renewLease.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15787) Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
[
https://issues.apache.org/jira/browse/HDFS-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296486#comment-17296486
]
Lisheng Sun commented on HDFS-15787:
Hi [~brahmareddy] [~cdouglas] Could you have time to confirm this
issue?[|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=cdouglas]
> Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
>
>
> Key: HDFS-15787
> URL: https://issues.apache.org/jira/browse/HDFS-15787
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-15787.001.patch, HDFS-15787.002.patch
>
>
> The method of FSNamesystem#internalReleaseLease() as follow:
>
> {code:java}
> boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
> String recoveryLeaseHolder) throws IOException {
> ...
> // Start recovery of the last block for this file
> // Only do so if there is no ongoing recovery for this block,
> // or the previous recovery for this block timed out.
> if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
> long blockRecoveryId = nextGenerationStamp(
> blockManager.isLegacyBlock(lastBlock));
> if(copyOnTruncate) {
> lastBlock.setGenerationStamp(blockRecoveryId);
> } else if(truncateRecovery) {
> recoveryBlock.setGenerationStamp(blockRecoveryId);
> }
> uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
> // Cannot close file right now, since the last block requires recovery.
> // This may potentially cause infinite loop in lease recovery
> // if there are no valid replicas on data-nodes.
> NameNode.stateChangeLog.warn(
> "DIR* NameSystem.internalReleaseLease: " +
> "File " + src + " has not been closed." +
> " Lease recovery is in progress. " +
> "RecoveryId = " + blockRecoveryId + " for block " + lastBlock);
> }
> lease = reassignLease(lease, src, recoveryLeaseHolder, pendingFile);
> leaseManager.renewLease(lease);
> break;
> }
> return false;
> }
> {code}
> Call LeaseManager#renewLease in
> FSNamesystem#reassignLease=>FSNamesystem#reassignLeaseInternal.
> So no need to call LeaseManager#renewLease again after
> leaseManager#renewLease.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15809) DeadNodeDetector doesn't remove live nodes from dead node set.
[ https://issues.apache.org/jira/browse/HDFS-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17292014#comment-17292014 ] Lisheng Sun commented on HDFS-15809: Hi [~LiJinglun] I think it is more appropriate to use LinkedHashSet. Its purpose is to ensure that the elements are unique and orderly. The safe mode is implemented by Collections.synchronized***(). > DeadNodeDetector doesn't remove live nodes from dead node set. > -- > > Key: HDFS-15809 > URL: https://issues.apache.org/jira/browse/HDFS-15809 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15809.001.patch, HDFS-15809.002.patch, > HDFS-15809.003.patch, HDFS-15809.004.patch > > > We found the dead node detector might never remove the alive nodes from the > dead node set in a big cluster. For example: > # 200 nodes are added to the dead node set by DeadNodeDetector. > # DeadNodeDetector#checkDeadNodes() adds 100 nodes to the > deadNodesProbeQueue because the queue limited length is 100. > # The probe threads start working and probe 30 nodes. > # DeadNodeDetector#checkDeadNodes() is scheduled again. It iterates the dead > node set and adds 30 nodes to the deadNodesProbeQueue. But the order is the > same as the last time. So the 30 nodes that has already been probed are added > to the queue again. > # Repeat 3 and 4. But we always add the first 30 nodes from the dead set. If > they are all dead then the live nodes behind them could never be recovered. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15809) DeadNodeDetector doesn't remove live nodes from dead node set.
[ https://issues.apache.org/jira/browse/HDFS-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287912#comment-17287912 ] Lisheng Sun commented on HDFS-15809: Hi [~LiJinglun] I understand the problem you want to solve is that the datanode in deadNodesProbeQueue is put in deadNodesProbeQueue. I think that the type of deadNodesProbeQueue is changed to LinkedHashSet solved this problem. No need to reconstruct a deduplication queue. Please help add a Unit Test that reproduces this problem. > DeadNodeDetector doesn't remove live nodes from dead node set. > -- > > Key: HDFS-15809 > URL: https://issues.apache.org/jira/browse/HDFS-15809 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15809.001.patch, HDFS-15809.002.patch, > HDFS-15809.003.patch > > > We found the dead node detector might never remove the alive nodes from the > dead node set in a big cluster. For example: > # 200 nodes are added to the dead node set by DeadNodeDetector. > # DeadNodeDetector#checkDeadNodes() adds 100 nodes to the > deadNodesProbeQueue because the queue limited length is 100. > # The probe threads start working and probe 30 nodes. > # DeadNodeDetector#checkDeadNodes() is scheduled again. It iterates the dead > node set and adds 30 nodes to the deadNodesProbeQueue. But the order is the > same as the last time. So the 30 nodes that has already been probed are added > to the queue again. > # Repeat 3 and 4. But we always add the first 30 nodes from the dead set. If > they are all dead then the live nodes behind them could never be recovered. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15787) Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
[
https://issues.apache.org/jira/browse/HDFS-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282881#comment-17282881
]
Lisheng Sun commented on HDFS-15787:
Hi [~elgoiri] [~ayushtkn] Could you mind to have another review? Thank you.
> Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
>
>
> Key: HDFS-15787
> URL: https://issues.apache.org/jira/browse/HDFS-15787
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-15787.001.patch, HDFS-15787.002.patch
>
>
> The method of FSNamesystem#internalReleaseLease() as follow:
>
> {code:java}
> boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
> String recoveryLeaseHolder) throws IOException {
> ...
> // Start recovery of the last block for this file
> // Only do so if there is no ongoing recovery for this block,
> // or the previous recovery for this block timed out.
> if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
> long blockRecoveryId = nextGenerationStamp(
> blockManager.isLegacyBlock(lastBlock));
> if(copyOnTruncate) {
> lastBlock.setGenerationStamp(blockRecoveryId);
> } else if(truncateRecovery) {
> recoveryBlock.setGenerationStamp(blockRecoveryId);
> }
> uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
> // Cannot close file right now, since the last block requires recovery.
> // This may potentially cause infinite loop in lease recovery
> // if there are no valid replicas on data-nodes.
> NameNode.stateChangeLog.warn(
> "DIR* NameSystem.internalReleaseLease: " +
> "File " + src + " has not been closed." +
> " Lease recovery is in progress. " +
> "RecoveryId = " + blockRecoveryId + " for block " + lastBlock);
> }
> lease = reassignLease(lease, src, recoveryLeaseHolder, pendingFile);
> leaseManager.renewLease(lease);
> break;
> }
> return false;
> }
> {code}
> Call LeaseManager#renewLease in
> FSNamesystem#reassignLease=>FSNamesystem#reassignLeaseInternal.
> So no need to call LeaseManager#renewLease again after
> leaseManager#renewLease.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15809) DeadNodeDetector doesn't remove live nodes from dead node set.
[
https://issues.apache.org/jira/browse/HDFS-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282879#comment-17282879
]
Lisheng Sun commented on HDFS-15809:
hi [~LiJinglun] for catching this scene.
{quote}
DeadNodeDetector#checkDeadNodes() is scheduled again. It iterates the dead node
set and adds 30 nodes to the deadNodesProbeQueue. But the order is the same as
the last time. So the 30 nodes that has already been probed are added to the
queue again.
{quote}
It may happen only when a large number of nodes hang up in a very large
cluster.
But the probability of this situation should very small. Because of the
background thread has been detecting the deadnode node, not just when call
checkdeadnode.
The solution I thought of is:
1、 Firstly shuffle all deadnodes in deadnode , poll fixed number of nodes into
deadNodesProbeQueue. This can avoid taking the same node every time
2、Adjust the queue size according to the cluster size.
I don’t understand your solution about this issue. Could you describe your
solution first? Thank you.
> DeadNodeDetector doesn't remove live nodes from dead node set.
> --
>
> Key: HDFS-15809
> URL: https://issues.apache.org/jira/browse/HDFS-15809
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HDFS-15809.001.patch
>
>
> We found the dead node detector might never remove the alive nodes from the
> dead node set in a big cluster. For example:
> # 200 nodes are added to the dead node set by DeadNodeDetector.
> # DeadNodeDetector#checkDeadNodes() adds 100 nodes to the
> deadNodesProbeQueue because the queue limited length is 100.
> # The probe threads start working and probe 30 nodes.
> # DeadNodeDetector#checkDeadNodes() is scheduled again. It iterates the dead
> node set and adds 30 nodes to the deadNodesProbeQueue. But the order is the
> same as the last time. So the 30 nodes that has already been probed are added
> to the queue again.
> # Repeat 3 and 4. But we always add the first 30 nodes from the dead set. If
> they are all dead then the live nodes behind them could never be recovered.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15787) Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
[
https://issues.apache.org/jira/browse/HDFS-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-15787:
---
Attachment: HDFS-15787.002.patch
> Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
>
>
> Key: HDFS-15787
> URL: https://issues.apache.org/jira/browse/HDFS-15787
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-15787.001.patch, HDFS-15787.002.patch
>
>
> The method of FSNamesystem#internalReleaseLease() as follow:
>
> {code:java}
> boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
> String recoveryLeaseHolder) throws IOException {
> ...
> // Start recovery of the last block for this file
> // Only do so if there is no ongoing recovery for this block,
> // or the previous recovery for this block timed out.
> if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
> long blockRecoveryId = nextGenerationStamp(
> blockManager.isLegacyBlock(lastBlock));
> if(copyOnTruncate) {
> lastBlock.setGenerationStamp(blockRecoveryId);
> } else if(truncateRecovery) {
> recoveryBlock.setGenerationStamp(blockRecoveryId);
> }
> uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
> // Cannot close file right now, since the last block requires recovery.
> // This may potentially cause infinite loop in lease recovery
> // if there are no valid replicas on data-nodes.
> NameNode.stateChangeLog.warn(
> "DIR* NameSystem.internalReleaseLease: " +
> "File " + src + " has not been closed." +
> " Lease recovery is in progress. " +
> "RecoveryId = " + blockRecoveryId + " for block " + lastBlock);
> }
> lease = reassignLease(lease, src, recoveryLeaseHolder, pendingFile);
> leaseManager.renewLease(lease);
> break;
> }
> return false;
> }
> {code}
> Call LeaseManager#renewLease in
> FSNamesystem#reassignLease=>FSNamesystem#reassignLeaseInternal.
> So no need to call LeaseManager#renewLease again after
> leaseManager#renewLease.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15787) Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
[
https://issues.apache.org/jira/browse/HDFS-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17275830#comment-17275830
]
Lisheng Sun commented on HDFS-15787:
Thanx [~hexiaoqiao] for your remind.
I think we can add a judgement to whether the leaseHolder is null. If it is
null, call LeaseManager#renewLease
I understand that in most cases the leaseHolder is't null, so we don't need to
call it twice.
> Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
>
>
> Key: HDFS-15787
> URL: https://issues.apache.org/jira/browse/HDFS-15787
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-15787.001.patch
>
>
> The method of FSNamesystem#internalReleaseLease() as follow:
>
> {code:java}
> boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
> String recoveryLeaseHolder) throws IOException {
> ...
> // Start recovery of the last block for this file
> // Only do so if there is no ongoing recovery for this block,
> // or the previous recovery for this block timed out.
> if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
> long blockRecoveryId = nextGenerationStamp(
> blockManager.isLegacyBlock(lastBlock));
> if(copyOnTruncate) {
> lastBlock.setGenerationStamp(blockRecoveryId);
> } else if(truncateRecovery) {
> recoveryBlock.setGenerationStamp(blockRecoveryId);
> }
> uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
> // Cannot close file right now, since the last block requires recovery.
> // This may potentially cause infinite loop in lease recovery
> // if there are no valid replicas on data-nodes.
> NameNode.stateChangeLog.warn(
> "DIR* NameSystem.internalReleaseLease: " +
> "File " + src + " has not been closed." +
> " Lease recovery is in progress. " +
> "RecoveryId = " + blockRecoveryId + " for block " + lastBlock);
> }
> lease = reassignLease(lease, src, recoveryLeaseHolder, pendingFile);
> leaseManager.renewLease(lease);
> break;
> }
> return false;
> }
> {code}
> Call LeaseManager#renewLease in
> FSNamesystem#reassignLease=>FSNamesystem#reassignLeaseInternal.
> So no need to call LeaseManager#renewLease again after
> leaseManager#renewLease.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15787) Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
[
https://issues.apache.org/jira/browse/HDFS-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17274120#comment-17274120
]
Lisheng Sun commented on HDFS-15787:
The failed UTs are unreated to this patch.
Hi [~ayushtkn] [~elgoiri] Could you have time to help review this patch？
> Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
>
>
> Key: HDFS-15787
> URL: https://issues.apache.org/jira/browse/HDFS-15787
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-15787.001.patch
>
>
> The method of FSNamesystem#internalReleaseLease() as follow:
>
> {code:java}
> boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
> String recoveryLeaseHolder) throws IOException {
> ...
> // Start recovery of the last block for this file
> // Only do so if there is no ongoing recovery for this block,
> // or the previous recovery for this block timed out.
> if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
> long blockRecoveryId = nextGenerationStamp(
> blockManager.isLegacyBlock(lastBlock));
> if(copyOnTruncate) {
> lastBlock.setGenerationStamp(blockRecoveryId);
> } else if(truncateRecovery) {
> recoveryBlock.setGenerationStamp(blockRecoveryId);
> }
> uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
> // Cannot close file right now, since the last block requires recovery.
> // This may potentially cause infinite loop in lease recovery
> // if there are no valid replicas on data-nodes.
> NameNode.stateChangeLog.warn(
> "DIR* NameSystem.internalReleaseLease: " +
> "File " + src + " has not been closed." +
> " Lease recovery is in progress. " +
> "RecoveryId = " + blockRecoveryId + " for block " + lastBlock);
> }
> lease = reassignLease(lease, src, recoveryLeaseHolder, pendingFile);
> leaseManager.renewLease(lease);
> break;
> }
> return false;
> }
> {code}
> Call LeaseManager#renewLease in
> FSNamesystem#reassignLease=>FSNamesystem#reassignLeaseInternal.
> So no need to call LeaseManager#renewLease again after
> leaseManager#renewLease.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15661) The DeadNodeDetector shouldn't be shared by different DFSClients.
[ https://issues.apache.org/jira/browse/HDFS-15661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-15661: --- Fix Version/s: 3.4.0 Resolution: Fixed Status: Resolved (was: Patch Available) > The DeadNodeDetector shouldn't be shared by different DFSClients. > - > > Key: HDFS-15661 > URL: https://issues.apache.org/jira/browse/HDFS-15661 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15661.001.patch, HDFS-15661.002.patch, > HDFS-15661.003.patch, HDFS-15661.004.patch, HDFS-15661.005.patch > > > Currently the DeadNodeDetector is a member of ClientContext. That means it is > shared by many different DFSClients. When one DFSClient.close() is invoked, > the DeadNodeDetecotor thread would be interrupted and impact other DFSClients. > From the original design of HDFS-13571 we could see the DeadNodeDetector is > supposed to share dead nodes of many input streams from the same client. > We should move the DeadNodeDetector as a member of DFSClient instead of > ClientContext. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15661) The DeadNodeDetector shouldn't be shared by different DFSClients.
[ https://issues.apache.org/jira/browse/HDFS-15661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273259#comment-17273259 ] Lisheng Sun commented on HDFS-15661: Commited to trunk. Thanks [~LiJinglun] for your report and contribution! Thanks [~weichiu] for your review! > The DeadNodeDetector shouldn't be shared by different DFSClients. > - > > Key: HDFS-15661 > URL: https://issues.apache.org/jira/browse/HDFS-15661 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15661.001.patch, HDFS-15661.002.patch, > HDFS-15661.003.patch, HDFS-15661.004.patch, HDFS-15661.005.patch > > > Currently the DeadNodeDetector is a member of ClientContext. That means it is > shared by many different DFSClients. When one DFSClient.close() is invoked, > the DeadNodeDetecotor thread would be interrupted and impact other DFSClients. > From the original design of HDFS-13571 we could see the DeadNodeDetector is > supposed to share dead nodes of many input streams from the same client. > We should move the DeadNodeDetector as a member of DFSClient instead of > ClientContext. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15661) The DeadNodeDetector shouldn't be shared by different DFSClients.
[ https://issues.apache.org/jira/browse/HDFS-15661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271382#comment-17271382 ] Lisheng Sun edited comment on HDFS-15661 at 1/25/21, 3:51 PM: -- +1 on the v005 patch. was (Author: leosun08): +1 on the v005.patch. > The DeadNodeDetector shouldn't be shared by different DFSClients. > - > > Key: HDFS-15661 > URL: https://issues.apache.org/jira/browse/HDFS-15661 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15661.001.patch, HDFS-15661.002.patch, > HDFS-15661.003.patch, HDFS-15661.004.patch, HDFS-15661.005.patch > > > Currently the DeadNodeDetector is a member of ClientContext. That means it is > shared by many different DFSClients. When one DFSClient.close() is invoked, > the DeadNodeDetecotor thread would be interrupted and impact other DFSClients. > From the original design of HDFS-13571 we could see the DeadNodeDetector is > supposed to share dead nodes of many input streams from the same client. > We should move the DeadNodeDetector as a member of DFSClient instead of > ClientContext. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15661) The DeadNodeDetector shouldn't be shared by different DFSClients.
[ https://issues.apache.org/jira/browse/HDFS-15661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271382#comment-17271382 ] Lisheng Sun edited comment on HDFS-15661 at 1/25/21, 3:51 PM: -- +1 on the v005.patch. was (Author: leosun08): +1 on https://issues.apache.org/jira/secure/attachment/13019288/HDFS-15661.005.patch > The DeadNodeDetector shouldn't be shared by different DFSClients. > - > > Key: HDFS-15661 > URL: https://issues.apache.org/jira/browse/HDFS-15661 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15661.001.patch, HDFS-15661.002.patch, > HDFS-15661.003.patch, HDFS-15661.004.patch, HDFS-15661.005.patch > > > Currently the DeadNodeDetector is a member of ClientContext. That means it is > shared by many different DFSClients. When one DFSClient.close() is invoked, > the DeadNodeDetecotor thread would be interrupted and impact other DFSClients. > From the original design of HDFS-13571 we could see the DeadNodeDetector is > supposed to share dead nodes of many input streams from the same client. > We should move the DeadNodeDetector as a member of DFSClient instead of > ClientContext. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15661) The DeadNodeDetector shouldn't be shared by different DFSClients.
[ https://issues.apache.org/jira/browse/HDFS-15661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271382#comment-17271382 ] Lisheng Sun commented on HDFS-15661: +1 on https://issues.apache.org/jira/secure/attachment/13019288/HDFS-15661.005.patch > The DeadNodeDetector shouldn't be shared by different DFSClients. > - > > Key: HDFS-15661 > URL: https://issues.apache.org/jira/browse/HDFS-15661 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15661.001.patch, HDFS-15661.002.patch, > HDFS-15661.003.patch, HDFS-15661.004.patch, HDFS-15661.005.patch > > > Currently the DeadNodeDetector is a member of ClientContext. That means it is > shared by many different DFSClients. When one DFSClient.close() is invoked, > the DeadNodeDetecotor thread would be interrupted and impact other DFSClients. > From the original design of HDFS-13571 we could see the DeadNodeDetector is > supposed to share dead nodes of many input streams from the same client. > We should move the DeadNodeDetector as a member of DFSClient instead of > ClientContext. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15661) The DeadNodeDetector shouldn't be shared by different DFSClients.
[ https://issues.apache.org/jira/browse/HDFS-15661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271048#comment-17271048 ] Lisheng Sun edited comment on HDFS-15661 at 1/25/21, 2:57 AM: -- Thank [~LiJinglun] for nice fixing，according to the way we commucate offline. Can you fix the checkstyle and confirm the failed ut is unrelated to the v04 patch? was (Author: leosun08): Thank [~LiJinglun] for nice fixing，according to the way we commucate offline. Can you fix the checkstyle and confirm the failed ut is unrelated to the v04 firstly ? > The DeadNodeDetector shouldn't be shared by different DFSClients. > - > > Key: HDFS-15661 > URL: https://issues.apache.org/jira/browse/HDFS-15661 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15661.001.patch, HDFS-15661.002.patch, > HDFS-15661.003.patch, HDFS-15661.004.patch > > > Currently the DeadNodeDetector is a member of ClientContext. That means it is > shared by many different DFSClients. When one DFSClient.close() is invoked, > the DeadNodeDetecotor thread would be interrupted and impact other DFSClients. > From the original design of HDFS-13571 we could see the DeadNodeDetector is > supposed to share dead nodes of many input streams from the same client. > We should move the DeadNodeDetector as a member of DFSClient instead of > ClientContext. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15661) The DeadNodeDetector shouldn't be shared by different DFSClients.
[ https://issues.apache.org/jira/browse/HDFS-15661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271048#comment-17271048 ] Lisheng Sun commented on HDFS-15661: Thank [~LiJinglun] for nice fixing，according to the way we commucate offline. Can you fix the checkstyle and confirm the failed ut is unrelated to the v04 firstly ? > The DeadNodeDetector shouldn't be shared by different DFSClients. > - > > Key: HDFS-15661 > URL: https://issues.apache.org/jira/browse/HDFS-15661 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15661.001.patch, HDFS-15661.002.patch, > HDFS-15661.003.patch, HDFS-15661.004.patch > > > Currently the DeadNodeDetector is a member of ClientContext. That means it is > shared by many different DFSClients. When one DFSClient.close() is invoked, > the DeadNodeDetecotor thread would be interrupted and impact other DFSClients. > From the original design of HDFS-13571 we could see the DeadNodeDetector is > supposed to share dead nodes of many input streams from the same client. > We should move the DeadNodeDetector as a member of DFSClient instead of > ClientContext. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15787) Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
[
https://issues.apache.org/jira/browse/HDFS-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-15787:
---
Attachment: HDFS-15787.001.patch
Status: Patch Available (was: Open)
> Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
>
>
> Key: HDFS-15787
> URL: https://issues.apache.org/jira/browse/HDFS-15787
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-15787.001.patch
>
>
> The method of FSNamesystem#internalReleaseLease() as follow:
>
> {code:java}
> boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
> String recoveryLeaseHolder) throws IOException {
> ...
> // Start recovery of the last block for this file
> // Only do so if there is no ongoing recovery for this block,
> // or the previous recovery for this block timed out.
> if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
> long blockRecoveryId = nextGenerationStamp(
> blockManager.isLegacyBlock(lastBlock));
> if(copyOnTruncate) {
> lastBlock.setGenerationStamp(blockRecoveryId);
> } else if(truncateRecovery) {
> recoveryBlock.setGenerationStamp(blockRecoveryId);
> }
> uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
> // Cannot close file right now, since the last block requires recovery.
> // This may potentially cause infinite loop in lease recovery
> // if there are no valid replicas on data-nodes.
> NameNode.stateChangeLog.warn(
> "DIR* NameSystem.internalReleaseLease: " +
> "File " + src + " has not been closed." +
> " Lease recovery is in progress. " +
> "RecoveryId = " + blockRecoveryId + " for block " + lastBlock);
> }
> lease = reassignLease(lease, src, recoveryLeaseHolder, pendingFile);
> leaseManager.renewLease(lease);
> break;
> }
> return false;
> }
> {code}
> Call LeaseManager#renewLease in
> FSNamesystem#reassignLease=>FSNamesystem#reassignLeaseInternal.
> So no need to call LeaseManager#renewLease again after
> leaseManager#renewLease.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15787) Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
[
https://issues.apache.org/jira/browse/HDFS-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-15787:
---
Description:
The method of FSNamesystem#internalReleaseLease() as follow:
{code:java}
boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
String recoveryLeaseHolder) throws IOException {
...
// Start recovery of the last block for this file
// Only do so if there is no ongoing recovery for this block,
// or the previous recovery for this block timed out.
if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
long blockRecoveryId = nextGenerationStamp(
blockManager.isLegacyBlock(lastBlock));
if(copyOnTruncate) {
lastBlock.setGenerationStamp(blockRecoveryId);
} else if(truncateRecovery) {
recoveryBlock.setGenerationStamp(blockRecoveryId);
}
uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
// Cannot close file right now, since the last block requires recovery.
// This may potentially cause infinite loop in lease recovery
// if there are no valid replicas on data-nodes.
NameNode.stateChangeLog.warn(
"DIR* NameSystem.internalReleaseLease: " +
"File " + src + " has not been closed." +
" Lease recovery is in progress. " +
"RecoveryId = " + blockRecoveryId + " for block " + lastBlock);
}
lease = reassignLease(lease, src, recoveryLeaseHolder, pendingFile);
leaseManager.renewLease(lease);
break;
}
return false;
}
{code}
Call LeaseManager#renewLease in
FSNamesystem#reassignLease=>FSNamesystem#reassignLeaseInternal.
So no need to call LeaseManager#renewLease again after
leaseManager#renewLease.
was:
The method of FSNamesystem#internalReleaseLease() as follow:
{code:java}
boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
String recoveryLeaseHolder) throws IOException {
...
// Start recovery of the last block for this file
// Only do so if there is no ongoing recovery for this block,
// or the previous recovery for this block timed out.
if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
long blockRecoveryId = nextGenerationStamp(
blockManager.isLegacyBlock(lastBlock));
if(copyOnTruncate) {
lastBlock.setGenerationStamp(blockRecoveryId);
} else if(truncateRecovery) {
recoveryBlock.setGenerationStamp(blockRecoveryId);
}
uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
// Cannot close file right now, since the last block requires recovery.
// This may potentially cause infinite loop in lease recovery
// if there are no valid replicas on data-nodes.
NameNode.stateChangeLog.warn(
"DIR* NameSystem.internalReleaseLease: " +
"File " + src + " has not been closed." +
" Lease recovery is in progress. " +
"RecoveryId = " + blockRecoveryId + " for block " + lastBlock);
}
lease = reassignLease(lease, src, recoveryLeaseHolder, pendingFile);
leaseManager.renewLease(lease);
break;
}
return false;
}
{code}
Call LeaseManager#renewLease in
FSNamesystem#reassignLease=>FSNamesystem#reassignLeaseInternal.
So no need to call LeaseManager#renewLease again after
leaseManager#renewLease.
> Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
>
>
> Key: HDFS-15787
> URL: https://issues.apache.org/jira/browse/HDFS-15787
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
>
> The method of FSNamesystem#internalReleaseLease() as follow:
>
> {code:java}
> boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
> String recoveryLeaseHolder) throws IOException {
> ...
> // Start recovery of the last block for this file
> // Only do so if there is no ongoing recovery for this block,
> // or the previous recovery for this block timed out.
> if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
> long blockRecoveryId = nextGenerationStamp(
> blockManager.isLegacyBlock(lastBlock));
> if(copyOnTruncate) {
> lastBlock.setGenerationStamp(blockRecoveryId);
> } else if(truncateRecovery) {
> recoveryBlock.setGenerationStamp(blockRecoveryId);
> }
> uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
> // Cannot close file right now, since the last block requires recovery.
> // This may potentially cause infinite loop in lease recovery
> // if there are no valid replicas on data-nodes.
> NameNode.stateChangeLog.warn(
> "DIR* NameSystem.internalReleaseLease: " +
>
[jira] [Updated] (HDFS-15787) Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
[
https://issues.apache.org/jira/browse/HDFS-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-15787:
---
Description:
The method of FSNamesystem#internalReleaseLease() as follow:
{code:java}
boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
String recoveryLeaseHolder) throws IOException {
...
// Start recovery of the last block for this file
// Only do so if there is no ongoing recovery for this block,
// or the previous recovery for this block timed out.
if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
long blockRecoveryId = nextGenerationStamp(
blockManager.isLegacyBlock(lastBlock));
if(copyOnTruncate) {
lastBlock.setGenerationStamp(blockRecoveryId);
} else if(truncateRecovery) {
recoveryBlock.setGenerationStamp(blockRecoveryId);
}
uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
// Cannot close file right now, since the last block requires recovery.
// This may potentially cause infinite loop in lease recovery
// if there are no valid replicas on data-nodes.
NameNode.stateChangeLog.warn(
"DIR* NameSystem.internalReleaseLease: " +
"File " + src + " has not been closed." +
" Lease recovery is in progress. " +
"RecoveryId = " + blockRecoveryId + " for block " + lastBlock);
}
lease = reassignLease(lease, src, recoveryLeaseHolder, pendingFile);
leaseManager.renewLease(lease);
break;
}
return false;
}
{code}
Call LeaseManager#renewLease in
FSNamesystem#reassignLease=>FSNamesystem#reassignLeaseInternal.
So no need to call LeaseManager#renewLease again after
leaseManager#renewLease.
was:
The method of FSNamesystem#internalReleaseLease() as follow:
{code:java}
boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
String recoveryLeaseHolder) throws IOException {
...
// Start recovery of the last block for this file
// Only do so if there is no ongoing recovery for this block,
// or the previous recovery for this block timed out.
if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
long blockRecoveryId = nextGenerationStamp(
blockManager.isLegacyBlock(lastBlock));
if(copyOnTruncate) {
lastBlock.setGenerationStamp(blockRecoveryId);
} else if(truncateRecovery) {
recoveryBlock.setGenerationStamp(blockRecoveryId);
}
uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
// Cannot close file right now, since the last block requires recovery.
// This may potentially cause infinite loop in lease recovery
// if there are no valid replicas on data-nodes.
NameNode.stateChangeLog.warn(
"DIR* NameSystem.internalReleaseLease: " +
"File " + src + " has not been closed." +
" Lease recovery is in progress. " +
"RecoveryId = " + blockRecoveryId + " for block " + lastBlock);
}
lease = reassignLease(lease, src, recoveryLeaseHolder, pendingFile);
leaseManager.renewLease(lease);
break;
}
return false;
}
{code}
> Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
>
>
> Key: HDFS-15787
> URL: https://issues.apache.org/jira/browse/HDFS-15787
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
>
> The method of FSNamesystem#internalReleaseLease() as follow:
>
> {code:java}
> boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
> String recoveryLeaseHolder) throws IOException {
> ...
> // Start recovery of the last block for this file
> // Only do so if there is no ongoing recovery for this block,
> // or the previous recovery for this block timed out.
> if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
> long blockRecoveryId = nextGenerationStamp(
> blockManager.isLegacyBlock(lastBlock));
> if(copyOnTruncate) {
> lastBlock.setGenerationStamp(blockRecoveryId);
> } else if(truncateRecovery) {
> recoveryBlock.setGenerationStamp(blockRecoveryId);
> }
> uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
> // Cannot close file right now, since the last block requires recovery.
> // This may potentially cause infinite loop in lease recovery
> // if there are no valid replicas on data-nodes.
> NameNode.stateChangeLog.warn(
> "DIR* NameSystem.internalReleaseLease: " +
> "File " + src + " has not been closed." +
> " Lease recovery is in progress. " +
> "RecoveryId = " + blockRecoveryId + " for block " + lastBlock);
>
[jira] [Created] (HDFS-15787) Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease
Lisheng Sun created HDFS-15787:
--
Summary: Remove unnecessary Lease Renew in
FSNamesystem#internalReleaseLease
Key: HDFS-15787
URL: https://issues.apache.org/jira/browse/HDFS-15787
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Lisheng Sun
Assignee: Lisheng Sun
The method of FSNamesystem#internalReleaseLease() as follow:
{code:java}
boolean internalReleaseLease(Lease lease, String src, INodesInPath iip,
String recoveryLeaseHolder) throws IOException {
...
// Start recovery of the last block for this file
// Only do so if there is no ongoing recovery for this block,
// or the previous recovery for this block timed out.
if (blockManager.addBlockRecoveryAttempt(lastBlock)) {
long blockRecoveryId = nextGenerationStamp(
blockManager.isLegacyBlock(lastBlock));
if(copyOnTruncate) {
lastBlock.setGenerationStamp(blockRecoveryId);
} else if(truncateRecovery) {
recoveryBlock.setGenerationStamp(blockRecoveryId);
}
uc.initializeBlockRecovery(lastBlock, blockRecoveryId, true);
// Cannot close file right now, since the last block requires recovery.
// This may potentially cause infinite loop in lease recovery
// if there are no valid replicas on data-nodes.
NameNode.stateChangeLog.warn(
"DIR* NameSystem.internalReleaseLease: " +
"File " + src + " has not been closed." +
" Lease recovery is in progress. " +
"RecoveryId = " + blockRecoveryId + " for block " + lastBlock);
}
lease = reassignLease(lease, src, recoveryLeaseHolder, pendingFile);
leaseManager.renewLease(lease);
break;
}
return false;
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15605) DeadNodeDetector supports getting deadnode from NameNode.
[ https://issues.apache.org/jira/browse/HDFS-15605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216172#comment-17216172 ] Lisheng Sun edited comment on HDFS-15605 at 10/19/20, 2:51 AM: --- Hi [~LiJinglun] Sorry，I relay to your mesage now. I understand that your improvment is that datatnode state completely depends on the namenode. But there are some problems which is why i did not choose to obtain the datanode status from the namenode. # The datanode status in namenode is not real time and the default is update the status without a heartbeat for 10 mins. At the same time there will be a delay in periodic calls to getDatanodeReport PRC. So the client may not get correct status of the datanode. In this cycle, all dfssinputstreams of the same client will read the dead node that has not be updated status in time. The delay is unacceptable for your online hhase cluster? # getDatanodeReport is a heavy call, and big clueter presssure on namnode. Can we solve the problem as you said by adjusting the elimation strategy on the client? When a dead datanode is found first time or continuous batch of dead nodes are found, it need to be confirmed again. was (Author: leosun08): Hi [~LiJinglun] Sorry，I relay to your mesage now. I understand that your improvment is that datatnode state completely depends on the namenode. But there are some problems which is why i did not choose to obtain the datanode status from the namenode. # The datanode status in namenode is not real time and the default is update the status without a heartbeat for 10 mins. At the same time there will be a delay in periodic calls to getDatanodeReport PRC. So the client may not get correct status of the datanode. In this cycle, all dfssinputstreams of the same client will read the dead node that has not be updated status in time. The delay is unacceptable for your online hhase cluster? # getDatanodeReport is a heavy call, and big clueter presssure on namnode. Can we solve the problem as you said by adjusting the elimation strategy on the client? When datanode dead is found first time or continuous batch of nodes, it need to be confirmed again. > DeadNodeDetector supports getting deadnode from NameNode. > - > > Key: HDFS-15605 > URL: https://issues.apache.org/jira/browse/HDFS-15605 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15605.001.patch, HDFS-15605.002.patch, > HDFS-15605.003.patch > > > When we are using DeadNodeDetector, sometimes it marks too many nodes as dead > and cause the read failures. The DeadNodeDetector assumes all the > getDatanodeInfo rpcs failed to return in time are dead nodes. But actually > not. A client side error or a slow rpc in DataNode might be marked as dead > too. For example the client side delay of the rpcThreadPool might cause the > getDatanodeInfo rpcs timeout and adding many datanodes to the dead list. > We have a simple improvement for this: the NameNode already knows which > datanodes are dead. So just update the dead list from NameNode using > DFSClient.datanodeReport(). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15605) DeadNodeDetector supports getting deadnode from NameNode.
[ https://issues.apache.org/jira/browse/HDFS-15605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216172#comment-17216172 ] Lisheng Sun commented on HDFS-15605: Hi [~LiJinglun] Sorry，I relay to your mesage now. I understand that your improvment is that datatnode state completely depends on the namenode. But there are some problems which is why i did not choose to obtain the datanode status from the namenode. # The datanode status in namenode is not real time and the default is update the status without a heartbeat for 10 mins. At the same time there will be a delay in periodic calls to getDatanodeReport PRC. So the client may not get correct status of the datanode. In this cycle, all dfssinputstreams of the same client will read the dead node that has not be updated status in time. The delay is unacceptable for your online hhase cluster? # getDatanodeReport is a heavy call, and big clueter presssure on namnode. Can we solve the problem as you said by adjusting the elimation strategy on the client? When datanode dead is found first time or continuous batch of nodes, it need to be confirmed again. > DeadNodeDetector supports getting deadnode from NameNode. > - > > Key: HDFS-15605 > URL: https://issues.apache.org/jira/browse/HDFS-15605 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15605.001.patch, HDFS-15605.002.patch, > HDFS-15605.003.patch > > > When we are using DeadNodeDetector, sometimes it marks too many nodes as dead > and cause the read failures. The DeadNodeDetector assumes all the > getDatanodeInfo rpcs failed to return in time are dead nodes. But actually > not. A client side error or a slow rpc in DataNode might be marked as dead > too. For example the client side delay of the rpcThreadPool might cause the > getDatanodeInfo rpcs timeout and adding many datanodes to the dead list. > We have a simple improvement for this: the NameNode already knows which > datanodes are dead. So just update the dead list from NameNode using > DFSClient.datanodeReport(). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15559) Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables
[
https://issues.apache.org/jira/browse/HDFS-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195812#comment-17195812
]
Lisheng Sun commented on HDFS-15559:
Hi [~hexiaoqiao]
I confirm the failed UTs are ok at local. And they are not related to this
patch. Thank you.
> Complement initialize member variables in
> TestHdfsConfigFields#initializeMemberVariables
>
>
> Key: HDFS-15559
> URL: https://issues.apache.org/jira/browse/HDFS-15559
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Minor
> Attachments: HDFS-15559.001.patch, HDFS-15559.002.patch
>
>
> There are some missing constant interfaces in
> TestHdfsConfigFields#initializeMemberVariables
> {code:java}
> @Override
> public void initializeMemberVariables() {
> xmlFilename = new String("hdfs-default.xml");
> configurationClasses = new Class[] { HdfsClientConfigKeys.class,
> HdfsClientConfigKeys.Failover.class,
> HdfsClientConfigKeys.StripedRead.class, DFSConfigKeys.class,
> HdfsClientConfigKeys.BlockWrite.class,
> HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.class };
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15573) Only log warning if considerLoad and considerStorageType are both true
[ https://issues.apache.org/jira/browse/HDFS-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17194212#comment-17194212 ] Lisheng Sun commented on HDFS-15573: Thanks [~sodonnell] for fixing this problem. [^HDFS-15573.001.patch] +1 LGFM. > Only log warning if considerLoad and considerStorageType are both true > -- > > Key: HDFS-15573 > URL: https://issues.apache.org/jira/browse/HDFS-15573 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.3.1, 3.4.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Attachments: HDFS-15573.001.patch > > > When we implemented HDFS-15255, we added a log message to warn if both > dfs.namenode.read.considerLoad and dfs.namenode.read.considerStorageType were > set to true, as they cannot be used together. > Somehow, we failed to wrap the log message in an IF statement, so it is > always printed incorrectly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15559) Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables
[
https://issues.apache.org/jira/browse/HDFS-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17194121#comment-17194121
]
Lisheng Sun commented on HDFS-15559:
Thanks [~hexiaoqiao]
The v002 patch rebased.
> Complement initialize member variables in
> TestHdfsConfigFields#initializeMemberVariables
>
>
> Key: HDFS-15559
> URL: https://issues.apache.org/jira/browse/HDFS-15559
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Minor
> Attachments: HDFS-15559.001.patch, HDFS-15559.002.patch
>
>
> There are some missing constant interfaces in
> TestHdfsConfigFields#initializeMemberVariables
> {code:java}
> @Override
> public void initializeMemberVariables() {
> xmlFilename = new String("hdfs-default.xml");
> configurationClasses = new Class[] { HdfsClientConfigKeys.class,
> HdfsClientConfigKeys.Failover.class,
> HdfsClientConfigKeys.StripedRead.class, DFSConfigKeys.class,
> HdfsClientConfigKeys.BlockWrite.class,
> HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.class };
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15559) Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables
[
https://issues.apache.org/jira/browse/HDFS-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-15559:
---
Attachment: HDFS-15559.002.patch
> Complement initialize member variables in
> TestHdfsConfigFields#initializeMemberVariables
>
>
> Key: HDFS-15559
> URL: https://issues.apache.org/jira/browse/HDFS-15559
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Minor
> Attachments: HDFS-15559.001.patch, HDFS-15559.002.patch
>
>
> There are some missing constant interfaces in
> TestHdfsConfigFields#initializeMemberVariables
> {code:java}
> @Override
> public void initializeMemberVariables() {
> xmlFilename = new String("hdfs-default.xml");
> configurationClasses = new Class[] { HdfsClientConfigKeys.class,
> HdfsClientConfigKeys.Failover.class,
> HdfsClientConfigKeys.StripedRead.class, DFSConfigKeys.class,
> HdfsClientConfigKeys.BlockWrite.class,
> HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.class };
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15559) Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables
[
https://issues.apache.org/jira/browse/HDFS-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193386#comment-17193386
]
Lisheng Sun edited comment on HDFS-15559 at 9/10/20, 5:56 AM:
--
Hi [~hexiaoqiao]
There are two places of missing infomation.
1. missing config in hdfs-default.xml
2. missing constant interfaces in
TestHdfsConfigFields#initializeMemberVariables as follow.
{code:java}
HdfsClientConfigKeys.Write.class,
HdfsClientConfigKeys.Read.class, HdfsClientConfigKeys.Retry.class,
HdfsClientConfigKeys.ShortCircuit.class,
HdfsClientConfigKeys.Mmap.class, HdfsClientConfigKeys.HedgedRead.class,
{code}
Since missing HdfsClientConfigKeys.Write.class in
TestHdfsConfigFields#initializeMemberVariables,
TestHdfsConfigFields#testCompareConfigurationClassAgainstXml pass when missed
config at HDFS-14694
Other missing constant interfaces in
TestHdfsConfigFields#initializeMemberVariables are not related to HDFS-14694,
so i create this issue.
was (Author: leosun08):
Hi [~hexiaoqiao]
There are two places of missing infomation.
1. missing config in hdfs-default.xml
2. missing constant interfaces in
TestHdfsConfigFields#initializeMemberVariables as follow.
{code:java}
HdfsClientConfigKeys.Write.class,
HdfsClientConfigKeys.Read.class, HdfsClientConfigKeys.Retry.class,
HdfsClientConfigKeys.ShortCircuit.class,
HdfsClientConfigKeys.Mmap.class, HdfsClientConfigKeys.HedgedRead.class,
{code}
Since missing HdfsClientConfigKeys.Write.class in
TestHdfsConfigFields#initializeMemberVariables,
TestHdfsConfigFields#testCompareConfigurationClassAgainstXml pass when missed
config at HDFS-14694
Other missing constant interfaces in
TestHdfsConfigFields#initializeMemberVariables are not related to HDFS-14694,
so i create this issue.++
> Complement initialize member variables in
> TestHdfsConfigFields#initializeMemberVariables
>
>
> Key: HDFS-15559
> URL: https://issues.apache.org/jira/browse/HDFS-15559
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Minor
> Attachments: HDFS-15559.001.patch
>
>
> There are some missing constant interfaces in
> TestHdfsConfigFields#initializeMemberVariables
> {code:java}
> @Override
> public void initializeMemberVariables() {
> xmlFilename = new String("hdfs-default.xml");
> configurationClasses = new Class[] { HdfsClientConfigKeys.class,
> HdfsClientConfigKeys.Failover.class,
> HdfsClientConfigKeys.StripedRead.class, DFSConfigKeys.class,
> HdfsClientConfigKeys.BlockWrite.class,
> HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.class };
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15559) Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables
[
https://issues.apache.org/jira/browse/HDFS-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193386#comment-17193386
]
Lisheng Sun commented on HDFS-15559:
Hi [~hexiaoqiao]
There are two places of missing infomation.
1. missing config in hdfs-default.xml
2. missing constant interfaces in
TestHdfsConfigFields#initializeMemberVariables as follow.
{code:java}
HdfsClientConfigKeys.Write.class,
HdfsClientConfigKeys.Read.class, HdfsClientConfigKeys.Retry.class,
HdfsClientConfigKeys.ShortCircuit.class,
HdfsClientConfigKeys.Mmap.class, HdfsClientConfigKeys.HedgedRead.class,
{code}
Since missing HdfsClientConfigKeys.Write.class in
TestHdfsConfigFields#initializeMemberVariables,
TestHdfsConfigFields#testCompareConfigurationClassAgainstXml pass when missed
config at HDFS-14694
Other missing constant interfaces in
TestHdfsConfigFields#initializeMemberVariables are not related to HDFS-14694,
so i create this issue.++
> Complement initialize member variables in
> TestHdfsConfigFields#initializeMemberVariables
>
>
> Key: HDFS-15559
> URL: https://issues.apache.org/jira/browse/HDFS-15559
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Minor
> Attachments: HDFS-15559.001.patch
>
>
> There are some missing constant interfaces in
> TestHdfsConfigFields#initializeMemberVariables
> {code:java}
> @Override
> public void initializeMemberVariables() {
> xmlFilename = new String("hdfs-default.xml");
> configurationClasses = new Class[] { HdfsClientConfigKeys.class,
> HdfsClientConfigKeys.Failover.class,
> HdfsClientConfigKeys.StripedRead.class, DFSConfigKeys.class,
> HdfsClientConfigKeys.BlockWrite.class,
> HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.class };
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15559) Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables
[
https://issues.apache.org/jira/browse/HDFS-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-15559:
---
Attachment: HDFS-15559.001.patch
Status: Patch Available (was: Open)
> Complement initialize member variables in
> TestHdfsConfigFields#initializeMemberVariables
>
>
> Key: HDFS-15559
> URL: https://issues.apache.org/jira/browse/HDFS-15559
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Minor
> Attachments: HDFS-15559.001.patch
>
>
> There are some missing constant interfaces in
> TestHdfsConfigFields#initializeMemberVariables
> {code:java}
> @Override
> public void initializeMemberVariables() {
> xmlFilename = new String("hdfs-default.xml");
> configurationClasses = new Class[] { HdfsClientConfigKeys.class,
> HdfsClientConfigKeys.Failover.class,
> HdfsClientConfigKeys.StripedRead.class, DFSConfigKeys.class,
> HdfsClientConfigKeys.BlockWrite.class,
> HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.class };
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15559) Complement initialize member variables in TestHdfsConfigFields
Lisheng Sun created HDFS-15559:
--
Summary: Complement initialize member variables in
TestHdfsConfigFields
Key: HDFS-15559
URL: https://issues.apache.org/jira/browse/HDFS-15559
Project: Hadoop HDFS
Issue Type: Test
Reporter: Lisheng Sun
Assignee: Lisheng Sun
There are some missing constant interfaces in
TestHdfsConfigFields#initializeMemberVariables
{code:java}
@Override
public void initializeMemberVariables() {
xmlFilename = new String("hdfs-default.xml");
configurationClasses = new Class[] { HdfsClientConfigKeys.class,
HdfsClientConfigKeys.Failover.class,
HdfsClientConfigKeys.StripedRead.class, DFSConfigKeys.class,
HdfsClientConfigKeys.BlockWrite.class,
HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.class };
}{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15559) Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables
[
https://issues.apache.org/jira/browse/HDFS-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-15559:
---
Summary: Complement initialize member variables in
TestHdfsConfigFields#initializeMemberVariables (was: Complement initialize
member variables in TestHdfsConfigFields#)
> Complement initialize member variables in
> TestHdfsConfigFields#initializeMemberVariables
>
>
> Key: HDFS-15559
> URL: https://issues.apache.org/jira/browse/HDFS-15559
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Minor
>
> There are some missing constant interfaces in
> TestHdfsConfigFields#initializeMemberVariables
> {code:java}
> @Override
> public void initializeMemberVariables() {
> xmlFilename = new String("hdfs-default.xml");
> configurationClasses = new Class[] { HdfsClientConfigKeys.class,
> HdfsClientConfigKeys.Failover.class,
> HdfsClientConfigKeys.StripedRead.class, DFSConfigKeys.class,
> HdfsClientConfigKeys.BlockWrite.class,
> HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.class };
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15559) Complement initialize member variables in TestHdfsConfigFields#
[
https://issues.apache.org/jira/browse/HDFS-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-15559:
---
Summary: Complement initialize member variables in TestHdfsConfigFields#
(was: Complement initialize member variables in TestHdfsConfigFields)
> Complement initialize member variables in TestHdfsConfigFields#
> ---
>
> Key: HDFS-15559
> URL: https://issues.apache.org/jira/browse/HDFS-15559
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Minor
>
> There are some missing constant interfaces in
> TestHdfsConfigFields#initializeMemberVariables
> {code:java}
> @Override
> public void initializeMemberVariables() {
> xmlFilename = new String("hdfs-default.xml");
> configurationClasses = new Class[] { HdfsClientConfigKeys.class,
> HdfsClientConfigKeys.Failover.class,
> HdfsClientConfigKeys.StripedRead.class, DFSConfigKeys.class,
> HdfsClientConfigKeys.BlockWrite.class,
> HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.class };
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191708#comment-17191708 ] Lisheng Sun commented on HDFS-14694: Thanx [~ayushtkn] for your **suggestion. I updated DFSOutputStream#recoverLease and uploaded the v014 patch. > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch, > HDFS-14694.009.patch, HDFS-14694.010.patch, HDFS-14694.011.patch, > HDFS-14694.012.patch, HDFS-14694.013.patch, HDFS-14694.014.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191708#comment-17191708 ] Lisheng Sun edited comment on HDFS-14694 at 9/7/20, 1:20 PM: - Thanx [~ayushtkn] for your suggestion. I updated DFSOutputStream#recoverLease and uploaded the v014 patch. was (Author: leosun08): Thanx [~ayushtkn] for your **suggestion. I updated DFSOutputStream#recoverLease and uploaded the v014 patch. > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch, > HDFS-14694.009.patch, HDFS-14694.010.patch, HDFS-14694.011.patch, > HDFS-14694.012.patch, HDFS-14694.013.patch, HDFS-14694.014.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14694: --- Attachment: HDFS-14694.014.patch > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch, > HDFS-14694.009.patch, HDFS-14694.010.patch, HDFS-14694.011.patch, > HDFS-14694.012.patch, HDFS-14694.013.patch, HDFS-14694.014.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14694: --- Attachment: HDFS-14694.013.patch > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch, > HDFS-14694.009.patch, HDFS-14694.010.patch, HDFS-14694.011.patch, > HDFS-14694.012.patch, HDFS-14694.013.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191422#comment-17191422 ] Lisheng Sun commented on HDFS-14694: Thanks [~ayushtkn] for patient review. According to your comment, I updated the patch and uploaded the v012 patch. > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch, > HDFS-14694.009.patch, HDFS-14694.010.patch, HDFS-14694.011.patch, > HDFS-14694.012.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14694: --- Attachment: HDFS-14694.012.patch > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch, > HDFS-14694.009.patch, HDFS-14694.010.patch, HDFS-14694.011.patch, > HDFS-14694.012.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190570#comment-17190570 ] Lisheng Sun commented on HDFS-14694: The failed UT is not related to this patch. > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch, > HDFS-14694.009.patch, HDFS-14694.010.patch, HDFS-14694.011.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15551) Tiny Improve for DeadNode detector
[ https://issues.apache.org/jira/browse/HDFS-15551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190537#comment-17190537 ] Lisheng Sun commented on HDFS-15551: Yeah，I would review it recently. > Tiny Improve for DeadNode detector > -- > > Key: HDFS-15551 > URL: https://issues.apache.org/jira/browse/HDFS-15551 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: 3.3.0 >Reporter: dark_num >Assignee: imbajin >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > # add or improve some logs for adding local & global deadnodes > # logic improve > # fix typo -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190479#comment-17190479 ] Lisheng Sun commented on HDFS-14694: Thanks [~hexiaoqiao] for patient review. The v011 patch removed unused code. > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch, > HDFS-14694.009.patch, HDFS-14694.010.patch, HDFS-14694.011.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14694: --- Attachment: HDFS-14694.011.patch > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch, > HDFS-14694.009.patch, HDFS-14694.010.patch, HDFS-14694.011.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14694: --- Attachment: (was: HDFS-14694.010.patch) > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch, > HDFS-14694.009.patch, HDFS-14694.010.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14694: --- Attachment: HDFS-14694.010.patch > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch, > HDFS-14694.009.patch, HDFS-14694.010.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14694: --- Attachment: HDFS-14694.010.patch > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch, > HDFS-14694.009.patch, HDFS-14694.010.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189164#comment-17189164 ] Lisheng Sun commented on HDFS-14694: Thanks [~hexiaoqiao] for your review. The failed UTs are ok in my local. And they are unrelated to this patch. The v009 patch fixed checkstyle. > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch, > HDFS-14694.009.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14694: --- Attachment: HDFS-14694.009.patch > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch, > HDFS-14694.009.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14694: --- Attachment: HDFS-14694.008.patch > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch, HDFS-14694.008.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187215#comment-17187215 ] Lisheng Sun commented on HDFS-14694: Thanks [~hexiaoqiao] for your review and good suggestion. I updated the patch and uploaded the v007 patch. The failded UT and asflicense is unrelated to this patch. Thank you. > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14694: --- Attachment: HDFS-14694.007.patch > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch, HDFS-14694.007.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186223#comment-17186223 ] Lisheng Sun commented on HDFS-14694: hi [~ayushtkn] [~weichiu] [~elgoiri] [~hexiaoqiao] Could you help review this patch? Thank you. > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15267) Implement Statistics Count for HttpFSFileSystem
[ https://issues.apache.org/jira/browse/HDFS-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185570#comment-17185570 ] Lisheng Sun edited comment on HDFS-15267 at 8/27/20, 2:55 AM: -- [~ayushtkn] >From the code point of view, HttpFSFileSystem is used as a proxy service, and >then forwards the request to the DistributedFileSystem. When request >HttpFSFileSystem every time, you will request DistributedFileSystem. So what >you see is double. was (Author: leosun08): >From the code point of view, HttpFSFileSystem is used as a proxy service, and >then forwards the request to the DistributedFileSystem. When request >HttpFSFileSystem every time, you will request DistributedFileSystem. So what >you see is double. > Implement Statistics Count for HttpFSFileSystem > --- > > Key: HDFS-15267 > URL: https://issues.apache.org/jira/browse/HDFS-15267 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15267.001.patch, HDFS-15267.002.patch > > > As of now, there is no count of ops maintained for HttpFSFileSystem like > DistributedFIleSysmtem & WebHDFSFilesystem -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15267) Implement Statistics Count for HttpFSFileSystem
[ https://issues.apache.org/jira/browse/HDFS-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185570#comment-17185570 ] Lisheng Sun commented on HDFS-15267: >From the code point of view, HttpFSFileSystem is used as a proxy service, and >then forwards the request to the DistributedFileSystem. When request >HttpFSFileSystem every time, you will request DistributedFileSystem. So what >you see is double. > Implement Statistics Count for HttpFSFileSystem > --- > > Key: HDFS-15267 > URL: https://issues.apache.org/jira/browse/HDFS-15267 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15267.001.patch, HDFS-15267.002.patch > > > As of now, there is no count of ops maintained for HttpFSFileSystem like > DistributedFIleSysmtem & WebHDFSFilesystem -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15267) Implement Statistics Count for HttpFSFileSystem
[ https://issues.apache.org/jira/browse/HDFS-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185075#comment-17185075 ] Lisheng Sun commented on HDFS-15267: the v002 patch trigger new check. > Implement Statistics Count for HttpFSFileSystem > --- > > Key: HDFS-15267 > URL: https://issues.apache.org/jira/browse/HDFS-15267 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15267.001.patch, HDFS-15267.002.patch > > > As of now, there is no count of ops maintained for HttpFSFileSystem like > DistributedFIleSysmtem & WebHDFSFilesystem -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15267) Implement Statistics Count for HttpFSFileSystem
[ https://issues.apache.org/jira/browse/HDFS-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-15267: --- Attachment: HDFS-15267.002.patch > Implement Statistics Count for HttpFSFileSystem > --- > > Key: HDFS-15267 > URL: https://issues.apache.org/jira/browse/HDFS-15267 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15267.001.patch, HDFS-15267.002.patch > > > As of now, there is no count of ops maintained for HttpFSFileSystem like > DistributedFIleSysmtem & WebHDFSFilesystem -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15267) Implement Statistics Count for HttpFSFileSystem
[ https://issues.apache.org/jira/browse/HDFS-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185016#comment-17185016 ] Lisheng Sun commented on HDFS-15267: hi [~ayushtkn] Are you still work on this jira? if not can I take over. > Implement Statistics Count for HttpFSFileSystem > --- > > Key: HDFS-15267 > URL: https://issues.apache.org/jira/browse/HDFS-15267 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15267.001.patch > > > As of now, there is no count of ops maintained for HttpFSFileSystem like > DistributedFIleSysmtem & WebHDFSFilesystem -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15267) Implement Statistics Count for HttpFSFileSystem
[ https://issues.apache.org/jira/browse/HDFS-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-15267: --- Attachment: HDFS-15267.001.patch Status: Patch Available (was: Open) > Implement Statistics Count for HttpFSFileSystem > --- > > Key: HDFS-15267 > URL: https://issues.apache.org/jira/browse/HDFS-15267 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15267.001.patch > > > As of now, there is no count of ops maintained for HttpFSFileSystem like > DistributedFIleSysmtem & WebHDFSFilesystem -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[
https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184986#comment-17184986
]
Lisheng Sun commented on HDFS-14694:
The failed UT is ok in my local.
{quote}TestHDFSContractMultipartUploader>AbstractContractMultipartUploaderTest.testConcurrentUploads:815
? IllegalArgument
{quote}
This failed UT is metioned in HDFS-15471 and this issue will fix it.
So All the failed UT is unrelated to this patch.
[~ayushtkn] Could you help review this patch? Thank you.
> Call recoverLease on DFSOutputStream close exception
>
>
> Key: HDFS-14694
> URL: https://issues.apache.org/jira/browse/HDFS-14694
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: Chen Zhang
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch,
> HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch,
> HDFS-14694.006.patch
>
>
> HDFS uses file-lease to manage opened files, when a file is not closed
> normally, NN will recover lease automatically after hard limit exceeded. But
> for a long running service(e.g. HBase), the hdfs-client will never die and NN
> don't have any chances to recover the file.
> Usually client program needs to handle exceptions by themself to avoid this
> condition(e.g. HBase automatically call recover lease for files that not
> closed normally), but in our experience, most services (in our company) don't
> process this condition properly, which will cause lots of files in abnormal
> status or even data loss.
> This Jira propose to add a feature that call recoverLease operation
> automatically when DFSOutputSteam close encounters exception. It should be
> disabled by default, but when somebody builds a long-running service based on
> HDFS, they can enable this option.
> We've add this feature to our internal Hadoop distribution for more than 3
> years, it's quite useful according our experience.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15471) TestHDFSContractMultipartUploader fails on trunk
[
https://issues.apache.org/jira/browse/HDFS-15471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun reassigned HDFS-15471:
--
Assignee: Lisheng Sun
> TestHDFSContractMultipartUploader fails on trunk
>
>
> Key: HDFS-15471
> URL: https://issues.apache.org/jira/browse/HDFS-15471
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Lisheng Sun
>Priority: Major
> Labels: test
>
> {{TestHDFSContractMultipartUploader}} fails on trunk with
> {{IllegalArgumentException}}
> {code:bash}
> [ERROR]
> testConcurrentUploads(org.apache.hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader)
> Time elapsed: 0.127 s <<< ERROR!
> java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:127)
> at
> org.apache.hadoop.test.LambdaTestUtils$ProportionalRetryInterval.(LambdaTestUtils.java:907)
> at
> org.apache.hadoop.fs.contract.AbstractContractMultipartUploaderTest.testConcurrentUploads(AbstractContractMultipartUploaderTest.java:815)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14694: --- Attachment: HDFS-14694.006.patch > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch, > HDFS-14694.006.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun reassigned HDFS-14694: -- Assignee: Lisheng Sun (was: Chen Zhang) > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183915#comment-17183915 ] Lisheng Sun commented on HDFS-14694: Hi [~zhangchen] Are you still working on this jira? If not, i will take over it. Hope you don't mind. > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14694: --- Attachment: HDFS-14694.005.patch > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch, HDFS-14694.005.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14694: --- Comment: was deleted (was: Hi [~zhangchen] Are you still working on this jira? If not, i will take over it. Hope you don't mind.) > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14694) Call recoverLease on DFSOutputStream close exception
[ https://issues.apache.org/jira/browse/HDFS-14694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183879#comment-17183879 ] Lisheng Sun commented on HDFS-14694: Hi [~zhangchen] Are you still working on this jira? If not, i will take over it. Hope you don't mind. > Call recoverLease on DFSOutputStream close exception > > > Key: HDFS-14694 > URL: https://issues.apache.org/jira/browse/HDFS-14694 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-14694.001.patch, HDFS-14694.002.patch, > HDFS-14694.003.patch, HDFS-14694.004.patch > > > HDFS uses file-lease to manage opened files, when a file is not closed > normally, NN will recover lease automatically after hard limit exceeded. But > for a long running service(e.g. HBase), the hdfs-client will never die and NN > don't have any chances to recover the file. > Usually client program needs to handle exceptions by themself to avoid this > condition(e.g. HBase automatically call recover lease for files that not > closed normally), but in our experience, most services (in our company) don't > process this condition properly, which will cause lots of files in abnormal > status or even data loss. > This Jira propose to add a feature that call recoverLease operation > automatically when DFSOutputSteam close encounters exception. It should be > disabled by default, but when somebody builds a long-running service based on > HDFS, they can enable this option. > We've add this feature to our internal Hadoop distribution for more than 3 > years, it's quite useful according our experience. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15409) Optimization Strategy for choosing ShortCircuitCache
[ https://issues.apache.org/jira/browse/HDFS-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149428#comment-17149428 ] Lisheng Sun edited comment on HDFS-15409 at 7/1/20, 1:10 PM: - Old (current) version shows better distribution when the clientShortCircuitNum is other num? was (Author: leosun08): Old (current) version shows better distribution when the clientShortCircuitNum is other nums? > Optimization Strategy for choosing ShortCircuitCache > - > > Key: HDFS-15409 > URL: https://issues.apache.org/jira/browse/HDFS-15409 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > > When clientShortCircuitNum is 10, the probability of falling into each > ShortCircuitCache is the same, while the probability of other > clientShortCircuitNum is different. > For example if clientShortCircuitNum is 3, when a lot of blockids of SSR are > ***1, ***4, ***7， this situation will fall into a ShortCircuitCache. > Since the real environment blockid is completely unpredictable, i think it is > need to design a strategy which is allocated to a specific ShortCircuitCache. > This should improve performance even more. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15409) Optimization Strategy for choosing ShortCircuitCache
[ https://issues.apache.org/jira/browse/HDFS-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149428#comment-17149428 ] Lisheng Sun commented on HDFS-15409: Old (current) version shows better distribution when the clientShortCircuitNum is other nums? > Optimization Strategy for choosing ShortCircuitCache > - > > Key: HDFS-15409 > URL: https://issues.apache.org/jira/browse/HDFS-15409 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > > When clientShortCircuitNum is 10, the probability of falling into each > ShortCircuitCache is the same, while the probability of other > clientShortCircuitNum is different. > For example if clientShortCircuitNum is 3, when a lot of blockids of SSR are > ***1, ***4, ***7， this situation will fall into a ShortCircuitCache. > Since the real environment blockid is completely unpredictable, i think it is > need to design a strategy which is allocated to a specific ShortCircuitCache. > This should improve performance even more. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15409) Optimization Strategy for choosing ShortCircuitCache
[
https://issues.apache.org/jira/browse/HDFS-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143743#comment-17143743
]
Lisheng Sun edited comment on HDFS-15409 at 6/24/20, 10:44 AM:
---
{quote}
So we could do it more tricky for case when clientShortCircuitNum = 3.
Divide by modulo 100 (instead of 10).
If two last digits between 0 and 32 then put it into clientShortCircuitNum[0]
-> 32% of blocks
If two last digits between 33 and 65 then put it into clientShortCircuitNum[1]
-> 32% of blocks
If two last digits between 66 and 99 then put it into clientShortCircuitNum[2]
-> 33% of blocks
Similar logic we can use when clientShortCircuitNum = 4:
Divide by modulo 100.
If two last digits between 0 and 24 then put it into clientShortCircuitNum[0]
-> 25% of blocks
If two last digits between 25 and 49 then put it into clientShortCircuitNum[1]
-> 25% of blocks
If two last digits between 50 and 74 then put it into clientShortCircuitNum[2]
-> 25% of blocks
If two last digits between 75 and 99 then put it into clientShortCircuitNum[3]
-> 25% of blocks
{quote}
i think these result don't depend on curent the code and use new strategy.
your test result is relatively uniform.
current strategy is blockid % 10.
so if the clientShortCircuitNum is 3 or 4 and according to the current code,
what is the result?
I think your new strategy result is better than current.:)
was (Author: leosun08):
{quote}
So we could do it more tricky for case when clientShortCircuitNum = 3.
Divide by modulo 100 (instead of 10).
If two last digits between 0 and 32 then put it into clientShortCircuitNum[0]
-> 32% of blocks
If two last digits between 33 and 65 then put it into clientShortCircuitNum[1]
-> 32% of blocks
If two last digits between 66 and 99 then put it into clientShortCircuitNum[2]
-> 33% of blocks
Similar logic we can use when clientShortCircuitNum = 4:
Divide by modulo 100.
If two last digits between 0 and 24 then put it into clientShortCircuitNum[0]
-> 25% of blocks
If two last digits between 25 and 49 then put it into clientShortCircuitNum[1]
-> 25% of blocks
If two last digits between 50 and 74 then put it into clientShortCircuitNum[2]
-> 25% of blocks
If two last digits between 75 and 99 then put it into clientShortCircuitNum[3]
-> 25% of blocks
{quote}
i think these result don't depend on curent the code and use new strategy.
your test result is relatively uniform.
current strategy is blockid % 10.
so if the clientShortCircuitNum is 3 or 4 and according to the current code,
what is the result?
> Optimization Strategy for choosing ShortCircuitCache
> -
>
> Key: HDFS-15409
> URL: https://issues.apache.org/jira/browse/HDFS-15409
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
>
> When clientShortCircuitNum is 10, the probability of falling into each
> ShortCircuitCache is the same, while the probability of other
> clientShortCircuitNum is different.
> For example if clientShortCircuitNum is 3, when a lot of blockids of SSR are
> ***1, ***4, ***7， this situation will fall into a ShortCircuitCache.
> Since the real environment blockid is completely unpredictable, i think it is
> need to design a strategy which is allocated to a specific ShortCircuitCache.
> This should improve performance even more.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15409) Optimization Strategy for choosing ShortCircuitCache
[
https://issues.apache.org/jira/browse/HDFS-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143743#comment-17143743
]
Lisheng Sun edited comment on HDFS-15409 at 6/24/20, 10:42 AM:
---
{quote}
So we could do it more tricky for case when clientShortCircuitNum = 3.
Divide by modulo 100 (instead of 10).
If two last digits between 0 and 32 then put it into clientShortCircuitNum[0]
-> 32% of blocks
If two last digits between 33 and 65 then put it into clientShortCircuitNum[1]
-> 32% of blocks
If two last digits between 66 and 99 then put it into clientShortCircuitNum[2]
-> 33% of blocks
Similar logic we can use when clientShortCircuitNum = 4:
Divide by modulo 100.
If two last digits between 0 and 24 then put it into clientShortCircuitNum[0]
-> 25% of blocks
If two last digits between 25 and 49 then put it into clientShortCircuitNum[1]
-> 25% of blocks
If two last digits between 50 and 74 then put it into clientShortCircuitNum[2]
-> 25% of blocks
If two last digits between 75 and 99 then put it into clientShortCircuitNum[3]
-> 25% of blocks
{quote}
i think these result don't depend on curent the code and use new strategy.
your test result is relatively uniform.
current strategy is blockid % 10.
so if the clientShortCircuitNum is 3 or 4 and according to the current code,
what is the result?
was (Author: leosun08):
{quote}
So we could do it more tricky for case when clientShortCircuitNum = 3.
Divide by modulo 100 (instead of 10).
If two last digits between 0 and 32 then put it into clientShortCircuitNum[0]
-> 32% of blocks
If two last digits between 33 and 65 then put it into clientShortCircuitNum[1]
-> 32% of blocks
If two last digits between 66 and 99 then put it into clientShortCircuitNum[2]
-> 33% of blocks
Similar logic we can use when clientShortCircuitNum = 4:
Divide by modulo 100.
If two last digits between 0 and 24 then put it into clientShortCircuitNum[0]
-> 25% of blocks
If two last digits between 25 and 49 then put it into clientShortCircuitNum[1]
-> 25% of blocks
If two last digits between 50 and 74 then put it into clientShortCircuitNum[2]
-> 25% of blocks
If two last digits between 75 and 99 then put it into clientShortCircuitNum[3]
-> 25% of blocks
{quote}
i think these result don't depend on curent the code and use new strategy.
current strategy is blockid % 10.
so if the clientShortCircuitNum is 3 or 4 and according to the current code,
what is the result?
> Optimization Strategy for choosing ShortCircuitCache
> -
>
> Key: HDFS-15409
> URL: https://issues.apache.org/jira/browse/HDFS-15409
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
>
> When clientShortCircuitNum is 10, the probability of falling into each
> ShortCircuitCache is the same, while the probability of other
> clientShortCircuitNum is different.
> For example if clientShortCircuitNum is 3, when a lot of blockids of SSR are
> ***1, ***4, ***7， this situation will fall into a ShortCircuitCache.
> Since the real environment blockid is completely unpredictable, i think it is
> need to design a strategy which is allocated to a specific ShortCircuitCache.
> This should improve performance even more.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15409) Optimization Strategy for choosing ShortCircuitCache
[
https://issues.apache.org/jira/browse/HDFS-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143743#comment-17143743
]
Lisheng Sun commented on HDFS-15409:
{quote}
So we could do it more tricky for case when clientShortCircuitNum = 3.
Divide by modulo 100 (instead of 10).
If two last digits between 0 and 32 then put it into clientShortCircuitNum[0]
-> 32% of blocks
If two last digits between 33 and 65 then put it into clientShortCircuitNum[1]
-> 32% of blocks
If two last digits between 66 and 99 then put it into clientShortCircuitNum[2]
-> 33% of blocks
Similar logic we can use when clientShortCircuitNum = 4:
Divide by modulo 100.
If two last digits between 0 and 24 then put it into clientShortCircuitNum[0]
-> 25% of blocks
If two last digits between 25 and 49 then put it into clientShortCircuitNum[1]
-> 25% of blocks
If two last digits between 50 and 74 then put it into clientShortCircuitNum[2]
-> 25% of blocks
If two last digits between 75 and 99 then put it into clientShortCircuitNum[3]
-> 25% of blocks
{quote}
i think these result don't depend on curent the code and use new strategy.
current strategy is blockid % 10.
so if the clientShortCircuitNum is 3 or 4 and according to the current code,
what is the result?
> Optimization Strategy for choosing ShortCircuitCache
> -
>
> Key: HDFS-15409
> URL: https://issues.apache.org/jira/browse/HDFS-15409
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
>
> When clientShortCircuitNum is 10, the probability of falling into each
> ShortCircuitCache is the same, while the probability of other
> clientShortCircuitNum is different.
> For example if clientShortCircuitNum is 3, when a lot of blockids of SSR are
> ***1, ***4, ***7， this situation will fall into a ShortCircuitCache.
> Since the real environment blockid is completely unpredictable, i think it is
> need to design a strategy which is allocated to a specific ShortCircuitCache.
> This should improve performance even more.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15409) Optimization Strategy for choosing ShortCircuitCache
[ https://issues.apache.org/jira/browse/HDFS-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17141638#comment-17141638 ] Lisheng Sun commented on HDFS-15409: yeah, it should be that. My idea is also to learn from the way of hbase to split the region. I think this strategy should be both complicated and interesting. So we should also think about it, it’s best to organize our ideas on a doc. > Optimization Strategy for choosing ShortCircuitCache > - > > Key: HDFS-15409 > URL: https://issues.apache.org/jira/browse/HDFS-15409 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > > When clientShortCircuitNum is 10, the probability of falling into each > ShortCircuitCache is the same, while the probability of other > clientShortCircuitNum is different. > For example if clientShortCircuitNum is 3, when a lot of blockids of SSR are > ***1, ***4, ***7， this situation will fall into a ShortCircuitCache. > Since the real environment blockid is completely unpredictable, i think it is > need to design a strategy which is allocated to a specific ShortCircuitCache. > This should improve performance even more. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15409) Optimization Strategy for choosing ShortCircuitCache
[ https://issues.apache.org/jira/browse/HDFS-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun reassigned HDFS-15409: -- Assignee: Lisheng Sun > Optimization Strategy for choosing ShortCircuitCache > - > > Key: HDFS-15409 > URL: https://issues.apache.org/jira/browse/HDFS-15409 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > > When clientShortCircuitNum is 10, the probability of falling into each > ShortCircuitCache is the same, while the probability of other > clientShortCircuitNum is different. > For example if clientShortCircuitNum is 3, when a lot of blockids of SSR are > ***1, ***4, ***7， this situation will fall into a ShortCircuitCache. > Since the real environment blockid is completely unpredictable, i think it is > need to design a strategy which is allocated to a specific ShortCircuitCache. > This should improve performance even more. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15409) Optimization Strategy for choosing ShortCircuitCache
[ https://issues.apache.org/jira/browse/HDFS-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17141494#comment-17141494 ] Lisheng Sun commented on HDFS-15409: Thank [~pustota] for you test. I have some question for your test. Is the clientShortCircuitNum only 10 in your test? i understand the distribution is average when clientShortCircuitNum is 10. I think we should test when clientShortCircuitNum is 3 or 4 and other nums. > Optimization Strategy for choosing ShortCircuitCache > - > > Key: HDFS-15409 > URL: https://issues.apache.org/jira/browse/HDFS-15409 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lisheng Sun >Priority: Major > > When clientShortCircuitNum is 10, the probability of falling into each > ShortCircuitCache is the same, while the probability of other > clientShortCircuitNum is different. > For example if clientShortCircuitNum is 3, when a lot of blockids of SSR are > ***1, ***4, ***7， this situation will fall into a ShortCircuitCache. > Since the real environment blockid is completely unpredictable, i think it is > need to design a strategy which is allocated to a specific ShortCircuitCache. > This should improve performance even more. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14895) Define LOG instead of BlockPlacementPolicy.LOG in DatanodeDescriptor#chooseStorage4Block
[ https://issues.apache.org/jira/browse/HDFS-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14895: --- Resolution: Duplicate Status: Resolved (was: Patch Available) > Define LOG instead of BlockPlacementPolicy.LOG in > DatanodeDescriptor#chooseStorage4Block > > > Key: HDFS-14895 > URL: https://issues.apache.org/jira/browse/HDFS-14895 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14895.001.patch > > > There is a noisy log with BlockPlacementPolicy.LOG, it's too hard to debug > problem. Define LOG instead of it in DatanodeDescriptor#chooseStorage4Block. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15409) Optimization Strategy for choosing ShortCircuitCache
[ https://issues.apache.org/jira/browse/HDFS-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-15409: --- Description: When clientShortCircuitNum is 10, the probability of falling into each ShortCircuitCache is the same, while the probability of other clientShortCircuitNum is different. For example if clientShortCircuitNum is 3, when a lot of blockids of SSR are ***1, ***4, ***7， this situation will fall into a ShortCircuitCache. Since the real environment blockid is completely unpredictable, i think it is need to design a strategy which is allocated to a specific ShortCircuitCache. This should improve performance even more. was: When clientShortCircuitNum is 10, the probability of falling into each ShortCircuitCache is the same, while the probability of other clientShortCircuitNum is different. For example if clientShortCircuitNum is 3, when a lot of blockids of SSR are ***1, ***4, ***7， this situation will fall into a ShortCircuitCache. Since the real environment blockid is completely unpredictable, i think it is need to design a strategy which is allocated to a specific ShortCircuitCache. This should improve performance even more. > Optimization Strategy for choosing ShortCircuitCache > - > > Key: HDFS-15409 > URL: https://issues.apache.org/jira/browse/HDFS-15409 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lisheng Sun >Priority: Major > > When clientShortCircuitNum is 10, the probability of falling into each > ShortCircuitCache is the same, while the probability of other > clientShortCircuitNum is different. > For example if clientShortCircuitNum is 3, when a lot of blockids of SSR are > ***1, ***4, ***7， this situation will fall into a ShortCircuitCache. > Since the real environment blockid is completely unpredictable, i think it is > need to design a strategy which is allocated to a specific ShortCircuitCache. > This should improve performance even more. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15409) Optimization Strategy for choosing ShortCircuitCache
Lisheng Sun created HDFS-15409: -- Summary: Optimization Strategy for choosing ShortCircuitCache Key: HDFS-15409 URL: https://issues.apache.org/jira/browse/HDFS-15409 Project: Hadoop HDFS Issue Type: Improvement Reporter: Lisheng Sun When clientShortCircuitNum is 10, the probability of falling into each ShortCircuitCache is the same, while the probability of other clientShortCircuitNum is different. For example if clientShortCircuitNum is 3, when a lot of blockids of SSR are ***1, ***4, ***7， this situation will fall into a ShortCircuitCache. Since the real environment blockid is completely unpredictable, i think it is need to design a strategy which is allocated to a specific ShortCircuitCache. This should improve performance even more. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15373) Fix number of threads in IPCLoggerChannel#createParallelExecutor
[ https://issues.apache.org/jira/browse/HDFS-15373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115611#comment-17115611 ] Lisheng Sun commented on HDFS-15373: the v01 patch LGFM. > Fix number of threads in IPCLoggerChannel#createParallelExecutor > > > Key: HDFS-15373 > URL: https://issues.apache.org/jira/browse/HDFS-15373 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-15373-01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13639) SlotReleaser is not fast enough
[
https://issues.apache.org/jira/browse/HDFS-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111267#comment-17111267
]
Lisheng Sun commented on HDFS-13639:
||case|| ||
|Before Case|perf_before_improve_SlotReleaser.png|
|After Case|perf_after_improve_SlotReleaser.png|
We can clearly see that: after the patch, YCSB get qps throughput increased
about 17.8% ( math: (4200 qps - 3700 qps) / 3700qps ~13.5%). See
CPS-1-min/name=hbase/ycsb-test/get on data grash. The stress test could be
finished smoothly, the qps of allocating and releasing match.
The x axis indicates time and the y axis indicates latency of
**percentile/name=hbase/ycsb-test/get on data grash.
After the patch, the latency is not long with the pressure test time.
> SlotReleaser is not fast enough
> ---
>
> Key: HDFS-13639
> URL: https://issues.apache.org/jira/browse/HDFS-13639
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.4.0, 2.6.0, 3.0.2
> Environment: 1. YCSB:
> {color:#00} recordcount=20
> fieldcount=1
> fieldlength=1000
> operationcount=1000
>
> workload=com.yahoo.ycsb.workloads.CoreWorkload
>
> table=ycsb-test
> columnfamily=C
> readproportion=1
> updateproportion=0
> insertproportion=0
> scanproportion=0
>
> maxscanlength=0
> requestdistribution=zipfian
>
> # default
> readallfields=true
> writeallfields=true
> scanlengthdistribution=constan{color}
> {color:#00}2. datanode:{color}
> -Xmx2048m -Xms2048m -Xmn1024m -XX:MaxDirectMemorySize=1024m
> -XX:MaxPermSize=256m -Xloggc:$run_dir/stdout/datanode_gc_${start_time}.log
> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=$log_dir -XX:+PrintGCApplicationStoppedTime
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSParallelRemarkEnabled
> -XX:+CMSClassUnloadingEnabled -XX:CMSMaxAbortablePrecleanTime=1
> -XX:+CMSScavengeBeforeRemark -XX:+PrintPromotionFailure
> -XX:+CMSConcurrentMTEnabled -XX:+ExplicitGCInvokesConcurrent
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -verbose:gc -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps
> {color:#00}3. regionserver:{color}
> {color:#00}-Xmx10g -Xms10g -XX:MaxDirectMemorySize=10g
> -XX:MaxGCPauseMillis=150 -XX:MaxTenuringThreshold=2
> -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=5
> -Xloggc:$run_dir/stdout/regionserver_gc_${start_time}.log -Xss256k
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$log_dir -verbose:gc
> -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy
> -XX:+PrintTenuringDistribution -XX:+PrintSafepointStatistics
> -XX:PrintSafepointStatisticsCount=1 -XX:PrintFLSStatistics=1
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=128m
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=65
> -XX:+ParallelRefProcEnabled -XX:ConcGCThreads=4 -XX:ParallelGCThreads=16
> -XX:G1HeapRegionSize=32m -XX:G1MixedGCCountTarget=64
> -XX:G1OldCSetRegionThresholdPercent=5{color}
> {color:#00}block cache is disabled:{color}{color:#00}
> hbase.bucketcache.size
> 0.9
> {color}
>
>Reporter: Gang Xie
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-13639-2.4.diff, HDFS-13639.001.patch,
> HDFS-13639.002.patch, ShortCircuitCache_new_slotReleaser.diff,
> perf_after_improve_SlotReleaser.png, perf_before_improve_SlotReleaser.png
>
>
> When test the performance of the ShortCircuit Read of the HDFS with YCSB, we
> find that SlotReleaser of the ShortCircuitCache has some performance issue.
> The problem is that, the qps of the slot releasing could only reach to 1000+
> while the qps of the slot allocating is ~3000. This means that the replica
> info on datanode could not be released in time, which causes a lot of GCs and
> finally full GCs.
>
> The fireflame graph shows that SlotReleaser spends a lot of time to do domain
> socket connecting and throw/catching the exception when close the domain
> socket and its streams. It doesn't make any sense to do the connecting and
> closing each time. Each time when we connect to the domain socket, Datanode
> allocates a new thread to free the slot. There are a lot of initializing
> work, and it's costly. We need reuse the domain socket.
>
> After switch to reuse the domain socket(see diff attached), we get great
> improvement(see the perf):
> # without reusing the domain socket, the get qps of the YCSB getting worse
> and worse, and after about 45 mins, full GC starts. When we reuse the domain
> socket, no full GC found,
[jira] [Updated] (HDFS-13639) SlotReleaser is not fast enough
[
https://issues.apache.org/jira/browse/HDFS-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-13639:
---
Description:
When test the performance of the ShortCircuit Read of the HDFS with YCSB, we
find that SlotReleaser of the ShortCircuitCache has some performance issue. The
problem is that, the qps of the slot releasing could only reach to 1000+ while
the qps of the slot allocating is ~3000. This means that the replica info on
datanode could not be released in time, which causes a lot of GCs and finally
full GCs.
The fireflame graph shows that SlotReleaser spends a lot of time to do domain
socket connecting and throw/catching the exception when close the domain socket
and its streams. It doesn't make any sense to do the connecting and closing
each time. Each time when we connect to the domain socket, Datanode allocates a
new thread to free the slot. There are a lot of initializing work, and it's
costly. We need reuse the domain socket.
After switch to reuse the domain socket(see diff attached), we get great
improvement(see the perf):
# without reusing the domain socket, the get qps of the YCSB getting worse and
worse, and after about 45 mins, full GC starts. When we reuse the domain
socket, no full GC found, and the stress test could be finished smoothly, the
qps of allocating and releasing match.
# Due to the datanode young GC, without the improvement, the YCSB get qps is
even smaller than the one with the improvement, ~3700 VS ~4200.
was:
When test the performance of the ShortCircuit Read of the HDFS with YCSB, we
find that SlotReleaser of the ShortCircuitCache has some performance issue. The
problem is that, the qps of the slot releasing could only reach to 1000+ while
the qps of the slot allocating is ~3000. This means that the replica info on
datanode could not be released in time, which causes a lot of GCs and finally
full GCs.
The fireflame graph shows that SlotReleaser spends a lot of time to do domain
socket connecting and throw/catching the exception when close the domain socket
and its streams. It doesn't make any sense to do the connecting and closing
each time. Each time when we connect to the domain socket, Datanode allocates a
new thread to free the slot. There are a lot of initializing work, and it's
costly. We need reuse the domain socket.
After switch to reuse the domain socket(see diff attached), we get great
improvement(see the perf):
# without reusing the domain socket, the get qps of the YCSB getting worse and
worse, and after about 45 mins, full GC starts. When we reuse the domain
socket, no full GC found, and the stress test could be finished smoothly, the
qps of allocating and releasing match.
# Due to the datanode young GC, without the improvement, the YCSB get qps is
even smaller than the one with the improvement, ~3700 VS ~4200.
The diff is against 2.4, and I think this issue exists till latest version. I
doesn't have test env with 2.7 and higher version.
> SlotReleaser is not fast enough
> ---
>
> Key: HDFS-13639
> URL: https://issues.apache.org/jira/browse/HDFS-13639
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.4.0, 2.6.0, 3.0.2
> Environment: 1. YCSB:
> {color:#00} recordcount=20
> fieldcount=1
> fieldlength=1000
> operationcount=1000
>
> workload=com.yahoo.ycsb.workloads.CoreWorkload
>
> table=ycsb-test
> columnfamily=C
> readproportion=1
> updateproportion=0
> insertproportion=0
> scanproportion=0
>
> maxscanlength=0
> requestdistribution=zipfian
>
> # default
> readallfields=true
> writeallfields=true
> scanlengthdistribution=constan{color}
> {color:#00}2. datanode:{color}
> -Xmx2048m -Xms2048m -Xmn1024m -XX:MaxDirectMemorySize=1024m
> -XX:MaxPermSize=256m -Xloggc:$run_dir/stdout/datanode_gc_${start_time}.log
> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=$log_dir -XX:+PrintGCApplicationStoppedTime
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSParallelRemarkEnabled
> -XX:+CMSClassUnloadingEnabled -XX:CMSMaxAbortablePrecleanTime=1
> -XX:+CMSScavengeBeforeRemark -XX:+PrintPromotionFailure
> -XX:+CMSConcurrentMTEnabled -XX:+ExplicitGCInvokesConcurrent
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -verbose:gc -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps
> {color:#00}3. regionserver:{color}
> {color:#00}-Xmx10g -Xms10g -XX:MaxDirectMemorySize=10g
> -XX:MaxGCPauseMillis=150 -XX:MaxTenuringThreshold=2
> -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=5
> -Xloggc:$run_dir/stdout/regionserver_gc_${start_time}.log -Xss256k
> -XX:+HeapDumpOnOutOfMemoryError
[jira] [Updated] (HDFS-13639) SlotReleaser is not fast enough
[
https://issues.apache.org/jira/browse/HDFS-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-13639:
---
Attachment: (was: image-2020-05-17-23-53-41-115.png)
> SlotReleaser is not fast enough
> ---
>
> Key: HDFS-13639
> URL: https://issues.apache.org/jira/browse/HDFS-13639
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.4.0, 2.6.0, 3.0.2
> Environment: 1. YCSB:
> {color:#00} recordcount=20
> fieldcount=1
> fieldlength=1000
> operationcount=1000
>
> workload=com.yahoo.ycsb.workloads.CoreWorkload
>
> table=ycsb-test
> columnfamily=C
> readproportion=1
> updateproportion=0
> insertproportion=0
> scanproportion=0
>
> maxscanlength=0
> requestdistribution=zipfian
>
> # default
> readallfields=true
> writeallfields=true
> scanlengthdistribution=constan{color}
> {color:#00}2. datanode:{color}
> -Xmx2048m -Xms2048m -Xmn1024m -XX:MaxDirectMemorySize=1024m
> -XX:MaxPermSize=256m -Xloggc:$run_dir/stdout/datanode_gc_${start_time}.log
> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=$log_dir -XX:+PrintGCApplicationStoppedTime
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSParallelRemarkEnabled
> -XX:+CMSClassUnloadingEnabled -XX:CMSMaxAbortablePrecleanTime=1
> -XX:+CMSScavengeBeforeRemark -XX:+PrintPromotionFailure
> -XX:+CMSConcurrentMTEnabled -XX:+ExplicitGCInvokesConcurrent
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -verbose:gc -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps
> {color:#00}3. regionserver:{color}
> {color:#00}-Xmx10g -Xms10g -XX:MaxDirectMemorySize=10g
> -XX:MaxGCPauseMillis=150 -XX:MaxTenuringThreshold=2
> -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=5
> -Xloggc:$run_dir/stdout/regionserver_gc_${start_time}.log -Xss256k
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$log_dir -verbose:gc
> -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy
> -XX:+PrintTenuringDistribution -XX:+PrintSafepointStatistics
> -XX:PrintSafepointStatisticsCount=1 -XX:PrintFLSStatistics=1
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=128m
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=65
> -XX:+ParallelRefProcEnabled -XX:ConcGCThreads=4 -XX:ParallelGCThreads=16
> -XX:G1HeapRegionSize=32m -XX:G1MixedGCCountTarget=64
> -XX:G1OldCSetRegionThresholdPercent=5{color}
> {color:#00}block cache is disabled:{color}{color:#00}
> hbase.bucketcache.size
> 0.9
> {color}
>
>Reporter: Gang Xie
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-13639-2.4.diff, HDFS-13639.001.patch,
> HDFS-13639.002.patch, ShortCircuitCache_new_slotReleaser.diff,
> perf_after_improve_SlotReleaser.png, perf_before_improve_SlotReleaser.png
>
>
> When test the performance of the ShortCircuit Read of the HDFS with YCSB, we
> find that SlotReleaser of the ShortCircuitCache has some performance issue.
> The problem is that, the qps of the slot releasing could only reach to 1000+
> while the qps of the slot allocating is ~3000. This means that the replica
> info on datanode could not be released in time, which causes a lot of GCs and
> finally full GCs.
>
> The fireflame graph shows that SlotReleaser spends a lot of time to do domain
> socket connecting and throw/catching the exception when close the domain
> socket and its streams. It doesn't make any sense to do the connecting and
> closing each time. Each time when we connect to the domain socket, Datanode
> allocates a new thread to free the slot. There are a lot of initializing
> work, and it's costly. We need reuse the domain socket.
>
> After switch to reuse the domain socket(see diff attached), we get great
> improvement(see the perf):
> # without reusing the domain socket, the get qps of the YCSB getting worse
> and worse, and after about 45 mins, full GC starts. When we reuse the domain
> socket, no full GC found, and the stress test could be finished smoothly, the
> qps of allocating and releasing match.
> # Due to the datanode young GC, without the improvement, the YCSB get qps is
> even smaller than the one with the improvement, ~3700 VS ~4200.
> The diff is against 2.4, and I think this issue exists till latest version. I
> doesn't have test env with 2.7 and higher version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
[jira] [Commented] (HDFS-12288) Fix DataNode's xceiver count calculation
[ https://issues.apache.org/jira/browse/HDFS-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1777#comment-1777 ] Lisheng Sun commented on HDFS-12288: hi [~elgoiri] Should we commit this patch to trunk? Thank you. > Fix DataNode's xceiver count calculation > > > Key: HDFS-12288 > URL: https://issues.apache.org/jira/browse/HDFS-12288 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Lukas Majercak >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-12288.001.patch, HDFS-12288.002.patch, > HDFS-12288.003.patch, HDFS-12288.004.patch, HDFS-12288.005.patch, > HDFS-12288.006.patch, HDFS-12288.007.patch, HDFS-12288.008.patch > > > The problem with the ThreadGroup.activeCount() method is that the method is > only a very rough estimate, and in reality returns the total number of > threads in the thread group as opposed to the threads actually running. > In some DNs, we saw this to return 50~ for a long time, even though the > actual number of DataXceiver threads was next to none. > This is a big issue as we use the xceiverCount to make decisions on the NN > for choosing replication source DN or returning DNs to clients for R/W. > The plan is to reuse the DataNodeMetrics.dataNodeActiveXceiversCount value > which only accounts for actual number of DataXcevier threads currently > running and thus represents the load on the DN much better. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13639) SlotReleaser is not fast enough
[
https://issues.apache.org/jira/browse/HDFS-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-13639:
---
Status: Open (was: Patch Available)
> SlotReleaser is not fast enough
> ---
>
> Key: HDFS-13639
> URL: https://issues.apache.org/jira/browse/HDFS-13639
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.0.2, 2.6.0, 2.4.0
> Environment: 1. YCSB:
> {color:#00} recordcount=20
> fieldcount=1
> fieldlength=1000
> operationcount=1000
>
> workload=com.yahoo.ycsb.workloads.CoreWorkload
>
> table=ycsb-test
> columnfamily=C
> readproportion=1
> updateproportion=0
> insertproportion=0
> scanproportion=0
>
> maxscanlength=0
> requestdistribution=zipfian
>
> # default
> readallfields=true
> writeallfields=true
> scanlengthdistribution=constan{color}
> {color:#00}2. datanode:{color}
> -Xmx2048m -Xms2048m -Xmn1024m -XX:MaxDirectMemorySize=1024m
> -XX:MaxPermSize=256m -Xloggc:$run_dir/stdout/datanode_gc_${start_time}.log
> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=$log_dir -XX:+PrintGCApplicationStoppedTime
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSParallelRemarkEnabled
> -XX:+CMSClassUnloadingEnabled -XX:CMSMaxAbortablePrecleanTime=1
> -XX:+CMSScavengeBeforeRemark -XX:+PrintPromotionFailure
> -XX:+CMSConcurrentMTEnabled -XX:+ExplicitGCInvokesConcurrent
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -verbose:gc -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps
> {color:#00}3. regionserver:{color}
> {color:#00}-Xmx10g -Xms10g -XX:MaxDirectMemorySize=10g
> -XX:MaxGCPauseMillis=150 -XX:MaxTenuringThreshold=2
> -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=5
> -Xloggc:$run_dir/stdout/regionserver_gc_${start_time}.log -Xss256k
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$log_dir -verbose:gc
> -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy
> -XX:+PrintTenuringDistribution -XX:+PrintSafepointStatistics
> -XX:PrintSafepointStatisticsCount=1 -XX:PrintFLSStatistics=1
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=128m
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=65
> -XX:+ParallelRefProcEnabled -XX:ConcGCThreads=4 -XX:ParallelGCThreads=16
> -XX:G1HeapRegionSize=32m -XX:G1MixedGCCountTarget=64
> -XX:G1OldCSetRegionThresholdPercent=5{color}
> {color:#00}block cache is disabled:{color}{color:#00}
> hbase.bucketcache.size
> 0.9
> {color}
>
>Reporter: Gang Xie
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-13639-2.4.diff, HDFS-13639.001.patch,
> HDFS-13639.002.patch, ShortCircuitCache_new_slotReleaser.diff,
> image-2020-05-17-23-53-41-115.png, perf_after_improve_SlotReleaser.png,
> perf_before_improve_SlotReleaser.png
>
>
> When test the performance of the ShortCircuit Read of the HDFS with YCSB, we
> find that SlotReleaser of the ShortCircuitCache has some performance issue.
> The problem is that, the qps of the slot releasing could only reach to 1000+
> while the qps of the slot allocating is ~3000. This means that the replica
> info on datanode could not be released in time, which causes a lot of GCs and
> finally full GCs.
>
> The fireflame graph shows that SlotReleaser spends a lot of time to do domain
> socket connecting and throw/catching the exception when close the domain
> socket and its streams. It doesn't make any sense to do the connecting and
> closing each time. Each time when we connect to the domain socket, Datanode
> allocates a new thread to free the slot. There are a lot of initializing
> work, and it's costly. We need reuse the domain socket.
>
> After switch to reuse the domain socket(see diff attached), we get great
> improvement(see the perf):
> # without reusing the domain socket, the get qps of the YCSB getting worse
> and worse, and after about 45 mins, full GC starts. When we reuse the domain
> socket, no full GC found, and the stress test could be finished smoothly, the
> qps of allocating and releasing match.
> # Due to the datanode young GC, without the improvement, the YCSB get qps is
> even smaller than the one with the improvement, ~3700 VS ~4200.
> The diff is against 2.4, and I think this issue exists till latest version. I
> doesn't have test env with 2.7 and higher version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
[jira] [Work started] (HDFS-13639) SlotReleaser is not fast enough
[
https://issues.apache.org/jira/browse/HDFS-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-13639 started by Lisheng Sun.
--
> SlotReleaser is not fast enough
> ---
>
> Key: HDFS-13639
> URL: https://issues.apache.org/jira/browse/HDFS-13639
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.4.0, 2.6.0, 3.0.2
> Environment: 1. YCSB:
> {color:#00} recordcount=20
> fieldcount=1
> fieldlength=1000
> operationcount=1000
>
> workload=com.yahoo.ycsb.workloads.CoreWorkload
>
> table=ycsb-test
> columnfamily=C
> readproportion=1
> updateproportion=0
> insertproportion=0
> scanproportion=0
>
> maxscanlength=0
> requestdistribution=zipfian
>
> # default
> readallfields=true
> writeallfields=true
> scanlengthdistribution=constan{color}
> {color:#00}2. datanode:{color}
> -Xmx2048m -Xms2048m -Xmn1024m -XX:MaxDirectMemorySize=1024m
> -XX:MaxPermSize=256m -Xloggc:$run_dir/stdout/datanode_gc_${start_time}.log
> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=$log_dir -XX:+PrintGCApplicationStoppedTime
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSParallelRemarkEnabled
> -XX:+CMSClassUnloadingEnabled -XX:CMSMaxAbortablePrecleanTime=1
> -XX:+CMSScavengeBeforeRemark -XX:+PrintPromotionFailure
> -XX:+CMSConcurrentMTEnabled -XX:+ExplicitGCInvokesConcurrent
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -verbose:gc -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps
> {color:#00}3. regionserver:{color}
> {color:#00}-Xmx10g -Xms10g -XX:MaxDirectMemorySize=10g
> -XX:MaxGCPauseMillis=150 -XX:MaxTenuringThreshold=2
> -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=5
> -Xloggc:$run_dir/stdout/regionserver_gc_${start_time}.log -Xss256k
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$log_dir -verbose:gc
> -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy
> -XX:+PrintTenuringDistribution -XX:+PrintSafepointStatistics
> -XX:PrintSafepointStatisticsCount=1 -XX:PrintFLSStatistics=1
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=128m
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=65
> -XX:+ParallelRefProcEnabled -XX:ConcGCThreads=4 -XX:ParallelGCThreads=16
> -XX:G1HeapRegionSize=32m -XX:G1MixedGCCountTarget=64
> -XX:G1OldCSetRegionThresholdPercent=5{color}
> {color:#00}block cache is disabled:{color}{color:#00}
> hbase.bucketcache.size
> 0.9
> {color}
>
>Reporter: Gang Xie
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-13639-2.4.diff, HDFS-13639.001.patch,
> HDFS-13639.002.patch, ShortCircuitCache_new_slotReleaser.diff,
> image-2020-05-17-23-53-41-115.png, perf_after_improve_SlotReleaser.png,
> perf_before_improve_SlotReleaser.png
>
>
> When test the performance of the ShortCircuit Read of the HDFS with YCSB, we
> find that SlotReleaser of the ShortCircuitCache has some performance issue.
> The problem is that, the qps of the slot releasing could only reach to 1000+
> while the qps of the slot allocating is ~3000. This means that the replica
> info on datanode could not be released in time, which causes a lot of GCs and
> finally full GCs.
>
> The fireflame graph shows that SlotReleaser spends a lot of time to do domain
> socket connecting and throw/catching the exception when close the domain
> socket and its streams. It doesn't make any sense to do the connecting and
> closing each time. Each time when we connect to the domain socket, Datanode
> allocates a new thread to free the slot. There are a lot of initializing
> work, and it's costly. We need reuse the domain socket.
>
> After switch to reuse the domain socket(see diff attached), we get great
> improvement(see the perf):
> # without reusing the domain socket, the get qps of the YCSB getting worse
> and worse, and after about 45 mins, full GC starts. When we reuse the domain
> socket, no full GC found, and the stress test could be finished smoothly, the
> qps of allocating and releasing match.
> # Due to the datanode young GC, without the improvement, the YCSB get qps is
> even smaller than the one with the improvement, ~3700 VS ~4200.
> The diff is against 2.4, and I think this issue exists till latest version. I
> doesn't have test env with 2.7 and higher version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
[jira] [Updated] (HDFS-13639) SlotReleaser is not fast enough
[
https://issues.apache.org/jira/browse/HDFS-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-13639:
---
Attachment: image-2020-05-17-23-53-41-115.png
> SlotReleaser is not fast enough
> ---
>
> Key: HDFS-13639
> URL: https://issues.apache.org/jira/browse/HDFS-13639
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.4.0, 2.6.0, 3.0.2
> Environment: 1. YCSB:
> {color:#00} recordcount=20
> fieldcount=1
> fieldlength=1000
> operationcount=1000
>
> workload=com.yahoo.ycsb.workloads.CoreWorkload
>
> table=ycsb-test
> columnfamily=C
> readproportion=1
> updateproportion=0
> insertproportion=0
> scanproportion=0
>
> maxscanlength=0
> requestdistribution=zipfian
>
> # default
> readallfields=true
> writeallfields=true
> scanlengthdistribution=constan{color}
> {color:#00}2. datanode:{color}
> -Xmx2048m -Xms2048m -Xmn1024m -XX:MaxDirectMemorySize=1024m
> -XX:MaxPermSize=256m -Xloggc:$run_dir/stdout/datanode_gc_${start_time}.log
> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=$log_dir -XX:+PrintGCApplicationStoppedTime
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSParallelRemarkEnabled
> -XX:+CMSClassUnloadingEnabled -XX:CMSMaxAbortablePrecleanTime=1
> -XX:+CMSScavengeBeforeRemark -XX:+PrintPromotionFailure
> -XX:+CMSConcurrentMTEnabled -XX:+ExplicitGCInvokesConcurrent
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -verbose:gc -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps
> {color:#00}3. regionserver:{color}
> {color:#00}-Xmx10g -Xms10g -XX:MaxDirectMemorySize=10g
> -XX:MaxGCPauseMillis=150 -XX:MaxTenuringThreshold=2
> -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=5
> -Xloggc:$run_dir/stdout/regionserver_gc_${start_time}.log -Xss256k
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$log_dir -verbose:gc
> -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy
> -XX:+PrintTenuringDistribution -XX:+PrintSafepointStatistics
> -XX:PrintSafepointStatisticsCount=1 -XX:PrintFLSStatistics=1
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=128m
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=65
> -XX:+ParallelRefProcEnabled -XX:ConcGCThreads=4 -XX:ParallelGCThreads=16
> -XX:G1HeapRegionSize=32m -XX:G1MixedGCCountTarget=64
> -XX:G1OldCSetRegionThresholdPercent=5{color}
> {color:#00}block cache is disabled:{color}{color:#00}
> hbase.bucketcache.size
> 0.9
> {color}
>
>Reporter: Gang Xie
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-13639-2.4.diff, HDFS-13639.001.patch,
> HDFS-13639.002.patch, ShortCircuitCache_new_slotReleaser.diff,
> image-2020-05-17-23-53-41-115.png, perf_after_improve_SlotReleaser.png,
> perf_before_improve_SlotReleaser.png
>
>
> When test the performance of the ShortCircuit Read of the HDFS with YCSB, we
> find that SlotReleaser of the ShortCircuitCache has some performance issue.
> The problem is that, the qps of the slot releasing could only reach to 1000+
> while the qps of the slot allocating is ~3000. This means that the replica
> info on datanode could not be released in time, which causes a lot of GCs and
> finally full GCs.
>
> The fireflame graph shows that SlotReleaser spends a lot of time to do domain
> socket connecting and throw/catching the exception when close the domain
> socket and its streams. It doesn't make any sense to do the connecting and
> closing each time. Each time when we connect to the domain socket, Datanode
> allocates a new thread to free the slot. There are a lot of initializing
> work, and it's costly. We need reuse the domain socket.
>
> After switch to reuse the domain socket(see diff attached), we get great
> improvement(see the perf):
> # without reusing the domain socket, the get qps of the YCSB getting worse
> and worse, and after about 45 mins, full GC starts. When we reuse the domain
> socket, no full GC found, and the stress test could be finished smoothly, the
> qps of allocating and releasing match.
> # Due to the datanode young GC, without the improvement, the YCSB get qps is
> even smaller than the one with the improvement, ~3700 VS ~4200.
> The diff is against 2.4, and I think this issue exists till latest version. I
> doesn't have test env with 2.7 and higher version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
[jira] [Commented] (HDFS-13639) SlotReleaser is not fast enough
[
https://issues.apache.org/jira/browse/HDFS-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105030#comment-17105030
]
Lisheng Sun commented on HDFS-13639:
hi [~sodonnell] [~ayushtkn] Could you have time to help review it? Thank you.
> SlotReleaser is not fast enough
> ---
>
> Key: HDFS-13639
> URL: https://issues.apache.org/jira/browse/HDFS-13639
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.4.0, 2.6.0, 3.0.2
> Environment: 1. YCSB:
> {color:#00} recordcount=20
> fieldcount=1
> fieldlength=1000
> operationcount=1000
>
> workload=com.yahoo.ycsb.workloads.CoreWorkload
>
> table=ycsb-test
> columnfamily=C
> readproportion=1
> updateproportion=0
> insertproportion=0
> scanproportion=0
>
> maxscanlength=0
> requestdistribution=zipfian
>
> # default
> readallfields=true
> writeallfields=true
> scanlengthdistribution=constan{color}
> {color:#00}2. datanode:{color}
> -Xmx2048m -Xms2048m -Xmn1024m -XX:MaxDirectMemorySize=1024m
> -XX:MaxPermSize=256m -Xloggc:$run_dir/stdout/datanode_gc_${start_time}.log
> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=$log_dir -XX:+PrintGCApplicationStoppedTime
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSParallelRemarkEnabled
> -XX:+CMSClassUnloadingEnabled -XX:CMSMaxAbortablePrecleanTime=1
> -XX:+CMSScavengeBeforeRemark -XX:+PrintPromotionFailure
> -XX:+CMSConcurrentMTEnabled -XX:+ExplicitGCInvokesConcurrent
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -verbose:gc -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps
> {color:#00}3. regionserver:{color}
> {color:#00}-Xmx10g -Xms10g -XX:MaxDirectMemorySize=10g
> -XX:MaxGCPauseMillis=150 -XX:MaxTenuringThreshold=2
> -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=5
> -Xloggc:$run_dir/stdout/regionserver_gc_${start_time}.log -Xss256k
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$log_dir -verbose:gc
> -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy
> -XX:+PrintTenuringDistribution -XX:+PrintSafepointStatistics
> -XX:PrintSafepointStatisticsCount=1 -XX:PrintFLSStatistics=1
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=128m
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=65
> -XX:+ParallelRefProcEnabled -XX:ConcGCThreads=4 -XX:ParallelGCThreads=16
> -XX:G1HeapRegionSize=32m -XX:G1MixedGCCountTarget=64
> -XX:G1OldCSetRegionThresholdPercent=5{color}
> {color:#00}block cache is disabled:{color}{color:#00}
> hbase.bucketcache.size
> 0.9
> {color}
>
>Reporter: Gang Xie
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-13639-2.4.diff, HDFS-13639.001.patch,
> HDFS-13639.002.patch, ShortCircuitCache_new_slotReleaser.diff,
> perf_after_improve_SlotReleaser.png, perf_before_improve_SlotReleaser.png
>
>
> When test the performance of the ShortCircuit Read of the HDFS with YCSB, we
> find that SlotReleaser of the ShortCircuitCache has some performance issue.
> The problem is that, the qps of the slot releasing could only reach to 1000+
> while the qps of the slot allocating is ~3000. This means that the replica
> info on datanode could not be released in time, which causes a lot of GCs and
> finally full GCs.
>
> The fireflame graph shows that SlotReleaser spends a lot of time to do domain
> socket connecting and throw/catching the exception when close the domain
> socket and its streams. It doesn't make any sense to do the connecting and
> closing each time. Each time when we connect to the domain socket, Datanode
> allocates a new thread to free the slot. There are a lot of initializing
> work, and it's costly. We need reuse the domain socket.
>
> After switch to reuse the domain socket(see diff attached), we get great
> improvement(see the perf):
> # without reusing the domain socket, the get qps of the YCSB getting worse
> and worse, and after about 45 mins, full GC starts. When we reuse the domain
> socket, no full GC found, and the stress test could be finished smoothly, the
> qps of allocating and releasing match.
> # Due to the datanode young GC, without the improvement, the YCSB get qps is
> even smaller than the one with the improvement, ~3700 VS ~4200.
> The diff is against 2.4, and I think this issue exists till latest version. I
> doesn't have test env with 2.7 and higher version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
[jira] [Commented] (HDFS-15255) Consider StorageType when DatanodeManager#sortLocatedBlock()
[
https://issues.apache.org/jira/browse/HDFS-15255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103856#comment-17103856
]
Lisheng Sun commented on HDFS-15255:
{quote}
2. There was a conflict due to HDFS-14283 in TestDFSInputStream. You
contributed that change - could you check my change there looks fine and if so
I think this one is good to commit now.
{quote}
hi [~sodonnell] i have checked it and it looks good. i think we could commit
the v10 patch to trunk. Thank you.
> Consider StorageType when DatanodeManager#sortLocatedBlock()
>
>
> Key: HDFS-15255
> URL: https://issues.apache.org/jira/browse/HDFS-15255
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: HDFS-15255-findbugs-test.001.patch,
> HDFS-15255.001.patch, HDFS-15255.002.patch, HDFS-15255.003.patch,
> HDFS-15255.004.patch, HDFS-15255.005.patch, HDFS-15255.006.patch,
> HDFS-15255.007.patch, HDFS-15255.008.patch, HDFS-15255.009.patch,
> HDFS-15255.010.patch, experiment-find-bugs.001.patch
>
>
> When only one replica of a block is SDD, the others are HDD.
> When the client reads the data, the current logic is that it considers the
> distance between the client and the dn. I think it should also consider the
> StorageType of the replica. Priority to return fast StorageType node when the
> distance is same.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12288) Fix DataNode's xceiver count calculation
[ https://issues.apache.org/jira/browse/HDFS-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103843#comment-17103843 ] Lisheng Sun commented on HDFS-12288: hi [~elgoiri] i understands this patch resolved the issues that is DataNode's xceiver count rough calculation. [~shahrs87] is intended to solve the problem. > Fix DataNode's xceiver count calculation > > > Key: HDFS-12288 > URL: https://issues.apache.org/jira/browse/HDFS-12288 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Lukas Majercak >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-12288.001.patch, HDFS-12288.002.patch, > HDFS-12288.003.patch, HDFS-12288.004.patch, HDFS-12288.005.patch, > HDFS-12288.006.patch, HDFS-12288.007.patch, HDFS-12288.008.patch > > > The problem with the ThreadGroup.activeCount() method is that the method is > only a very rough estimate, and in reality returns the total number of > threads in the thread group as opposed to the threads actually running. > In some DNs, we saw this to return 50~ for a long time, even though the > actual number of DataXceiver threads was next to none. > This is a big issue as we use the xceiverCount to make decisions on the NN > for choosing replication source DN or returning DNs to clients for R/W. > The plan is to reuse the DataNodeMetrics.dataNodeActiveXceiversCount value > which only accounts for actual number of DataXcevier threads currently > running and thus represents the load on the DN much better. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14283) DFSInputStream to prefer cached replica
[
https://issues.apache.org/jira/browse/HDFS-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099836#comment-17099836
]
Lisheng Sun commented on HDFS-14283:
the v009 patch updated description in hdfs-default.xml.
> DFSInputStream to prefer cached replica
> ---
>
> Key: HDFS-14283
> URL: https://issues.apache.org/jira/browse/HDFS-14283
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.6.0
> Environment: HDFS Caching
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14283.001.patch, HDFS-14283.002.patch,
> HDFS-14283.003.patch, HDFS-14283.004.patch, HDFS-14283.005.patch,
> HDFS-14283.006.patch, HDFS-14283.007.patch, HDFS-14283.008.patch,
> HDFS-14283.009.patch
>
>
> HDFS Caching offers performance benefits. However, currently NameNode does
> not treat cached replica with higher priority, so HDFS caching is only useful
> when cache replication = 3, that is to say, all replicas are cached in
> memory, so that a client doesn't randomly pick an uncached replica.
> HDFS-6846 proposed to let NameNode give higher priority to cached replica.
> Changing a logic in NameNode is always tricky so that didn't get much
> traction. Here I propose a different approach: let client (DFSInputStream)
> prefer cached replica.
> A {{LocatedBlock}} object already contains cached replica location so a
> client has the needed information. I think we can change
> {{DFSInputStream#getBestNodeDNAddrPair()}} for this purpose.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14283) DFSInputStream to prefer cached replica
[
https://issues.apache.org/jira/browse/HDFS-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-14283:
---
Attachment: HDFS-14283.009.patch
> DFSInputStream to prefer cached replica
> ---
>
> Key: HDFS-14283
> URL: https://issues.apache.org/jira/browse/HDFS-14283
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.6.0
> Environment: HDFS Caching
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14283.001.patch, HDFS-14283.002.patch,
> HDFS-14283.003.patch, HDFS-14283.004.patch, HDFS-14283.005.patch,
> HDFS-14283.006.patch, HDFS-14283.007.patch, HDFS-14283.008.patch,
> HDFS-14283.009.patch
>
>
> HDFS Caching offers performance benefits. However, currently NameNode does
> not treat cached replica with higher priority, so HDFS caching is only useful
> when cache replication = 3, that is to say, all replicas are cached in
> memory, so that a client doesn't randomly pick an uncached replica.
> HDFS-6846 proposed to let NameNode give higher priority to cached replica.
> Changing a logic in NameNode is always tricky so that didn't get much
> traction. Here I propose a different approach: let client (DFSInputStream)
> prefer cached replica.
> A {{LocatedBlock}} object already contains cached replica location so a
> client has the needed information. I think we can change
> {{DFSInputStream#getBestNodeDNAddrPair()}} for this purpose.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14283) DFSInputStream to prefer cached replica
[
https://issues.apache.org/jira/browse/HDFS-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-14283:
---
Attachment: (was: HDFS-14283.009.patch)
> DFSInputStream to prefer cached replica
> ---
>
> Key: HDFS-14283
> URL: https://issues.apache.org/jira/browse/HDFS-14283
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.6.0
> Environment: HDFS Caching
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14283.001.patch, HDFS-14283.002.patch,
> HDFS-14283.003.patch, HDFS-14283.004.patch, HDFS-14283.005.patch,
> HDFS-14283.006.patch, HDFS-14283.007.patch, HDFS-14283.008.patch
>
>
> HDFS Caching offers performance benefits. However, currently NameNode does
> not treat cached replica with higher priority, so HDFS caching is only useful
> when cache replication = 3, that is to say, all replicas are cached in
> memory, so that a client doesn't randomly pick an uncached replica.
> HDFS-6846 proposed to let NameNode give higher priority to cached replica.
> Changing a logic in NameNode is always tricky so that didn't get much
> traction. Here I propose a different approach: let client (DFSInputStream)
> prefer cached replica.
> A {{LocatedBlock}} object already contains cached replica location so a
> client has the needed information. I think we can change
> {{DFSInputStream#getBestNodeDNAddrPair()}} for this purpose.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14283) DFSInputStream to prefer cached replica
[
https://issues.apache.org/jira/browse/HDFS-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-14283:
---
Attachment: HDFS-14283.009.patch
> DFSInputStream to prefer cached replica
> ---
>
> Key: HDFS-14283
> URL: https://issues.apache.org/jira/browse/HDFS-14283
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.6.0
> Environment: HDFS Caching
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14283.001.patch, HDFS-14283.002.patch,
> HDFS-14283.003.patch, HDFS-14283.004.patch, HDFS-14283.005.patch,
> HDFS-14283.006.patch, HDFS-14283.007.patch, HDFS-14283.008.patch,
> HDFS-14283.009.patch
>
>
> HDFS Caching offers performance benefits. However, currently NameNode does
> not treat cached replica with higher priority, so HDFS caching is only useful
> when cache replication = 3, that is to say, all replicas are cached in
> memory, so that a client doesn't randomly pick an uncached replica.
> HDFS-6846 proposed to let NameNode give higher priority to cached replica.
> Changing a logic in NameNode is always tricky so that didn't get much
> traction. Here I propose a different approach: let client (DFSInputStream)
> prefer cached replica.
> A {{LocatedBlock}} object already contains cached replica location so a
> client has the needed information. I think we can change
> {{DFSInputStream#getBestNodeDNAddrPair()}} for this purpose.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14283) DFSInputStream to prefer cached replica
[
https://issues.apache.org/jira/browse/HDFS-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099652#comment-17099652
]
Lisheng Sun commented on HDFS-14283:
Thanx [~ayushtkn] for your suggestion.
{quote}
+ if (!deadNodes.containsKey(cachedLocs[i])
For this can we use dfsClient.getDeadNodes(this).containsKey(nodes[i])? it is
added as part of DeadDatanodeDetection feature. If yes, May be we can refactor
the if checks into a single method and use at both places.
{quote}
the v008 patch fixed this problem.
{quote}
return new DNAddrPair(chosenNode, targetAddr, storageType, block);
storagaeType will be null if using cachedReplica, is it ok?
{quote}
storagaeType is only used when NameNode choose DataNode.
It will not be used when Client establishes a connection with DataNode.
See Send and DataXceiver#readBlock parameters：
{code:java}
public void readBlock(final ExtendedBlock block,
final Token blockToken,
final String clientName,
final long blockOffset,
final long length,
final boolean sendChecksum,
final CachingStrategy cachingStrategy) throws IOException {
{code}
> DFSInputStream to prefer cached replica
> ---
>
> Key: HDFS-14283
> URL: https://issues.apache.org/jira/browse/HDFS-14283
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.6.0
> Environment: HDFS Caching
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14283.001.patch, HDFS-14283.002.patch,
> HDFS-14283.003.patch, HDFS-14283.004.patch, HDFS-14283.005.patch,
> HDFS-14283.006.patch, HDFS-14283.007.patch, HDFS-14283.008.patch
>
>
> HDFS Caching offers performance benefits. However, currently NameNode does
> not treat cached replica with higher priority, so HDFS caching is only useful
> when cache replication = 3, that is to say, all replicas are cached in
> memory, so that a client doesn't randomly pick an uncached replica.
> HDFS-6846 proposed to let NameNode give higher priority to cached replica.
> Changing a logic in NameNode is always tricky so that didn't get much
> traction. Here I propose a different approach: let client (DFSInputStream)
> prefer cached replica.
> A {{LocatedBlock}} object already contains cached replica location so a
> client has the needed information. I think we can change
> {{DFSInputStream#getBestNodeDNAddrPair()}} for this purpose.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
