[jira] [Created] (HDFS-15444) mkdir should not create dir in fallback if the dir already in mount Path
Uma Maheswara Rao G created HDFS-15444: -- Summary: mkdir should not create dir in fallback if the dir already in mount Path Key: HDFS-15444 URL: https://issues.apache.org/jira/browse/HDFS-15444 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15430) create should work when parent dir is internalDir and fallback configured.
[ https://issues.apache.org/jira/browse/HDFS-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15430: --- Status: Patch Available (was: Open) > create should work when parent dir is internalDir and fallback configured. > --- > > Key: HDFS-15430 > URL: https://issues.apache.org/jira/browse/HDFS-15430 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > create will not work if the parent dir is Internal mount dir (non leaf in > mount path) and fall back configured. > Since fallback is available and if same tree structure available in fallback, > we should be able to create in fallback fs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15374) Add documentation for fedbalance tool
[ https://issues.apache.org/jira/browse/HDFS-15374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147510#comment-17147510 ] Jinglun commented on HDFS-15374: Hi [~linyiqun], shall we commit this ?:) > Add documentation for fedbalance tool > - > > Key: HDFS-15374 > URL: https://issues.apache.org/jira/browse/HDFS-15374 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: BalanceProcedureScheduler.png, > FedBalance_Screenshot1.jpg, FedBalance_Screenshot2.jpg, > FedBalance_Screenshot3.jpg, HDFS-15374.001.patch, HDFS-15374.002.patch, > HDFS-15374.003.patch, HDFS-15374.004.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15410) Add separated config file fedbalance-default.xml for fedbalance tool
[ https://issues.apache.org/jira/browse/HDFS-15410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147511#comment-17147511 ] Jinglun commented on HDFS-15410: Hi [~linyiqun], shall we commit this ?:) > Add separated config file fedbalance-default.xml for fedbalance tool > > > Key: HDFS-15410 > URL: https://issues.apache.org/jira/browse/HDFS-15410 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15410.001.patch, HDFS-15410.002.patch, > HDFS-15410.003.patch, HDFS-15410.004.patch > > > Add a separated config file named fedbalance-default.xml for fedbalance tool > configs. It's like the ditcp-default.xml for distcp tool. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15443) Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure.
[
https://issues.apache.org/jira/browse/HDFS-15443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
AMC-team updated HDFS-15443:
Description:
Configuration parameter dfs.datanode.max.transfer.threads is to specify the
maximum number of threads to use for transferring data in and out of the DN.
This is a vital param that need to tune carefully.
{code:java}
// DataXceiverServer.java
// Make sure the xceiver count is not exceeded
intcurXceiverCount = datanode.getXceiverCount();
if (curXceiverCount > maxXceiverCount) {
thrownewIOException("Xceiver count " + curXceiverCount
+ " exceeds the limit of concurrent xceivers: "
+ maxXceiverCount);
}
{code}
There are many issues that caused by not setting this param to an appropriate
value. However, there is no any check code to restrict the parameter. Although
having a hard-and-fast rule is difficult because we need to consider number of
cores, main memory etc, *we can prevent users from setting this value to an
absolute wrong value by accident.* (e.g. a negative value that totally break
the availability of datanode.)
*How to fix:*
Add proper check code for the parameter.
was:
Configuration parameter dfs.datanode.max.transfer.threads is to specify the
maximum number of threads to use for transferring data in and out of the DN.
This is a vital param that need to tune carefully. There are many bad cases
that caused by not setting this param to an appropriate value.
{code:java}
// DataXceiverServer.java
// Make sure the xceiver count is not exceeded
intcurXceiverCount = datanode.getXceiverCount();
if (curXceiverCount > maxXceiverCount) {
thrownewIOException("Xceiver count " + curXceiverCount
+ " exceeds the limit of concurrent xceivers: "
+ maxXceiverCount);
}
{code}
However, there is no any check code to restrict the parameter. Although having
a hard-and-fast rule is difficult because we need to consider number of cores,
main memory etc, *we should at least prevent users from setting this value to
an absolute wrong value by accident.* (e.g. a negative value that totally break
the availability of datanode.)
*How to fix:*
Add proper check code for the parameter.
> Setting dfs.datanode.max.transfer.threads to a very small value can cause
> strange failure.
> --
>
> Key: HDFS-15443
> URL: https://issues.apache.org/jira/browse/HDFS-15443
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: AMC-team
>Priority: Major
>
> Configuration parameter dfs.datanode.max.transfer.threads is to specify the
> maximum number of threads to use for transferring data in and out of the DN.
> This is a vital param that need to tune carefully.
> {code:java}
> // DataXceiverServer.java
> // Make sure the xceiver count is not exceeded
> intcurXceiverCount = datanode.getXceiverCount();
> if (curXceiverCount > maxXceiverCount) {
> thrownewIOException("Xceiver count " + curXceiverCount
> + " exceeds the limit of concurrent xceivers: "
> + maxXceiverCount);
> }
> {code}
> There are many issues that caused by not setting this param to an appropriate
> value. However, there is no any check code to restrict the parameter.
> Although having a hard-and-fast rule is difficult because we need to consider
> number of cores, main memory etc, *we can prevent users from setting this
> value to an absolute wrong value by accident.* (e.g. a negative value that
> totally break the availability of datanode.)
> *How to fix:*
> Add proper check code for the parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15443) Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure.
[
https://issues.apache.org/jira/browse/HDFS-15443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
AMC-team updated HDFS-15443:
Summary: Setting dfs.datanode.max.transfer.threads to a very small value
can cause strange failure. (was: Setting )
> Setting dfs.datanode.max.transfer.threads to a very small value can cause
> strange failure.
> --
>
> Key: HDFS-15443
> URL: https://issues.apache.org/jira/browse/HDFS-15443
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: AMC-team
>Priority: Major
>
> Configuration parameter dfs.datanode.max.transfer.threads is to specify the
> maximum number of threads to use for transferring data in and out of the DN.
> This is a vital param that need to tune carefully. There are many bad cases
> that caused by not setting this param to an appropriate value.
> {code:java}
> // DataXceiverServer.java
> // Make sure the xceiver count is not exceeded
> intcurXceiverCount = datanode.getXceiverCount();
> if (curXceiverCount > maxXceiverCount) {
> thrownewIOException("Xceiver count " + curXceiverCount
> + " exceeds the limit of concurrent xceivers: "
> + maxXceiverCount);
> }
> {code}
> However, there is no any check code to restrict the parameter. Although
> having a hard-and-fast rule is difficult because we need to consider number
> of cores, main memory etc, *we should at least prevent users from setting
> this value to an absolute wrong value by accident.* (e.g. a negative value
> that totally break the availability of datanode.)
> *How to fix:*
> Add proper check code for the parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15443) Setting
[
https://issues.apache.org/jira/browse/HDFS-15443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
AMC-team updated HDFS-15443:
Summary: Setting (was: dfs.datanode.max.transfer.threads should have
check code)
> Setting
>
>
> Key: HDFS-15443
> URL: https://issues.apache.org/jira/browse/HDFS-15443
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: AMC-team
>Priority: Major
>
> Configuration parameter dfs.datanode.max.transfer.threads is to specify the
> maximum number of threads to use for transferring data in and out of the DN.
> This is a vital param that need to tune carefully. There are many bad cases
> that caused by not setting this param to an appropriate value.
> {code:java}
> // DataXceiverServer.java
> // Make sure the xceiver count is not exceeded
> intcurXceiverCount = datanode.getXceiverCount();
> if (curXceiverCount > maxXceiverCount) {
> thrownewIOException("Xceiver count " + curXceiverCount
> + " exceeds the limit of concurrent xceivers: "
> + maxXceiverCount);
> }
> {code}
> However, there is no any check code to restrict the parameter. Although
> having a hard-and-fast rule is difficult because we need to consider number
> of cores, main memory etc, *we should at least prevent users from setting
> this value to an absolute wrong value by accident.* (e.g. a negative value
> that totally break the availability of datanode.)
> *How to fix:*
> Add proper check code for the parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15442) Image upload may fail if dfs.image.transfer.chunksize wrongly set to negative value
[
https://issues.apache.org/jira/browse/HDFS-15442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
AMC-team updated HDFS-15442:
Summary: Image upload may fail if dfs.image.transfer.chunksize wrongly set
to negative value (was: Image upload will fail if dfs.image.transfer.chunksize
wrongly set to negative value)
> Image upload may fail if dfs.image.transfer.chunksize wrongly set to negative
> value
> ---
>
> Key: HDFS-15442
> URL: https://issues.apache.org/jira/browse/HDFS-15442
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: AMC-team
>Priority: Major
>
> In current implementation of checkpoint image transfer, if the file length is
> bigger than the configured value dfs.image.transfer.chunksize, it will use
> chunked streaming mode to avoid internal buffering. This mode should be used
> only if more than chunkSize data is present to upload, otherwise upload may
> not happen sometimes.
> {code:java}
> //TransferFsImage.java
> int chunkSize = (int) conf.getLongBytes(
> DFSConfigKeys.DFS_IMAGE_TRANSFER_CHUNKSIZE_KEY,
> DFSConfigKeys.DFS_IMAGE_TRANSFER_CHUNKSIZE_DEFAULT);
> if (imageFile.length() > chunkSize) {
> // using chunked streaming mode to support upload of 2GB+ files and to
> // avoid internal buffering.
> // this mode should be used only if more than chunkSize data is present
> // to upload. otherwise upload may not happen sometimes.
> connection.setChunkedStreamingMode(chunkSize);
> }
> {code}
> There is no any check code for this parameter. User may accidentally set this
> value to a wrong value. Here, if the user set chunkSize to a negative value.
> Chunked streaming mode will always be used. In
> setChunkedStreamingMode(chunkSize), there is a correction code that if the
> chunkSize is <=0, it will be change to DEFAULT_CHUNK_SIZE.
> {code:java}
> public void setChunkedStreamingMode (int chunklen) {
> if (connected) {
> throw new IllegalStateException ("Can't set streaming mode: already
> connected");
> }
> if (fixedContentLength != -1 || fixedContentLengthLong != -1) {
> throw new IllegalStateException ("Fixed length streaming mode set");
> }
> chunkLength = chunklen <=0? DEFAULT_CHUNK_SIZE : chunklen;
> }
> {code}
> However, the correction may be too late.
> If the user set dfs.image.transfer.chunksize to value that <= 0, even for
> images whose imageFile.length() < DEFAULT_CHUNK_SIZE will use chunked
> streaming mode and may fail the upload as mentioned above. *(This scenario
> may not be common, but* *we can prevent users setting this param to an
> extremely small value.**)*
> *How to fix:*
> Add checking code or correction code right after parsing the config value
> before really use the value (if statement and setChunkedStreamingMode).
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15442) Image upload will fail if dfs.image.transfer.chunksize wrongly set to negative value
[
https://issues.apache.org/jira/browse/HDFS-15442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
AMC-team updated HDFS-15442:
Summary: Image upload will fail if dfs.image.transfer.chunksize wrongly set
to negative value (was: Image upload will fail if )
> Image upload will fail if dfs.image.transfer.chunksize wrongly set to
> negative value
>
>
> Key: HDFS-15442
> URL: https://issues.apache.org/jira/browse/HDFS-15442
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: AMC-team
>Priority: Major
>
> In current implementation of checkpoint image transfer, if the file length is
> bigger than the configured value dfs.image.transfer.chunksize, it will use
> chunked streaming mode to avoid internal buffering. This mode should be used
> only if more than chunkSize data is present to upload, otherwise upload may
> not happen sometimes.
> {code:java}
> //TransferFsImage.java
> int chunkSize = (int) conf.getLongBytes(
> DFSConfigKeys.DFS_IMAGE_TRANSFER_CHUNKSIZE_KEY,
> DFSConfigKeys.DFS_IMAGE_TRANSFER_CHUNKSIZE_DEFAULT);
> if (imageFile.length() > chunkSize) {
> // using chunked streaming mode to support upload of 2GB+ files and to
> // avoid internal buffering.
> // this mode should be used only if more than chunkSize data is present
> // to upload. otherwise upload may not happen sometimes.
> connection.setChunkedStreamingMode(chunkSize);
> }
> {code}
> There is no any check code for this parameter. User may accidentally set this
> value to a wrong value. Here, if the user set chunkSize to a negative value.
> Chunked streaming mode will always be used. In
> setChunkedStreamingMode(chunkSize), there is a correction code that if the
> chunkSize is <=0, it will be change to DEFAULT_CHUNK_SIZE.
> {code:java}
> public void setChunkedStreamingMode (int chunklen) {
> if (connected) {
> throw new IllegalStateException ("Can't set streaming mode: already
> connected");
> }
> if (fixedContentLength != -1 || fixedContentLengthLong != -1) {
> throw new IllegalStateException ("Fixed length streaming mode set");
> }
> chunkLength = chunklen <=0? DEFAULT_CHUNK_SIZE : chunklen;
> }
> {code}
> However, the correction may be too late.
> If the user set dfs.image.transfer.chunksize to value that <= 0, even for
> images whose imageFile.length() < DEFAULT_CHUNK_SIZE will use chunked
> streaming mode and may fail the upload as mentioned above. *(This scenario
> may not be common, but* *we can prevent users setting this param to an
> extremely small value.**)*
> *How to fix:*
> Add checking code or correction code right after parsing the config value
> before really use the value (if statement and setChunkedStreamingMode).
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15442) Image upload will fail if
[
https://issues.apache.org/jira/browse/HDFS-15442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
AMC-team updated HDFS-15442:
Summary: Image upload will fail if (was: dfs.image.transfer.chunksize
should have check code before use)
> Image upload will fail if
> --
>
> Key: HDFS-15442
> URL: https://issues.apache.org/jira/browse/HDFS-15442
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: AMC-team
>Priority: Major
>
> In current implementation of checkpoint image transfer, if the file length is
> bigger than the configured value dfs.image.transfer.chunksize, it will use
> chunked streaming mode to avoid internal buffering. This mode should be used
> only if more than chunkSize data is present to upload, otherwise upload may
> not happen sometimes.
> {code:java}
> //TransferFsImage.java
> int chunkSize = (int) conf.getLongBytes(
> DFSConfigKeys.DFS_IMAGE_TRANSFER_CHUNKSIZE_KEY,
> DFSConfigKeys.DFS_IMAGE_TRANSFER_CHUNKSIZE_DEFAULT);
> if (imageFile.length() > chunkSize) {
> // using chunked streaming mode to support upload of 2GB+ files and to
> // avoid internal buffering.
> // this mode should be used only if more than chunkSize data is present
> // to upload. otherwise upload may not happen sometimes.
> connection.setChunkedStreamingMode(chunkSize);
> }
> {code}
> There is no any check code for this parameter. User may accidentally set this
> value to a wrong value. Here, if the user set chunkSize to a negative value.
> Chunked streaming mode will always be used. In
> setChunkedStreamingMode(chunkSize), there is a correction code that if the
> chunkSize is <=0, it will be change to DEFAULT_CHUNK_SIZE.
> {code:java}
> public void setChunkedStreamingMode (int chunklen) {
> if (connected) {
> throw new IllegalStateException ("Can't set streaming mode: already
> connected");
> }
> if (fixedContentLength != -1 || fixedContentLengthLong != -1) {
> throw new IllegalStateException ("Fixed length streaming mode set");
> }
> chunkLength = chunklen <=0? DEFAULT_CHUNK_SIZE : chunklen;
> }
> {code}
> However, the correction may be too late.
> If the user set dfs.image.transfer.chunksize to value that <= 0, even for
> images whose imageFile.length() < DEFAULT_CHUNK_SIZE will use chunked
> streaming mode and may fail the upload as mentioned above. *(This scenario
> may not be common, but* *we can prevent users setting this param to an
> extremely small value.**)*
> *How to fix:*
> Add checking code or correction code right after parsing the config value
> before really use the value (if statement and setChunkedStreamingMode).
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15440) The doc of dfs.disk.balancer.block.tolerance.percent is misleading
[
https://issues.apache.org/jira/browse/HDFS-15440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
AMC-team updated HDFS-15440:
Summary: The doc of dfs.disk.balancer.block.tolerance.percent is misleading
(was: The using of dfs.disk.balancer.block.tolerance.percent is inconsistent
with doc)
> The doc of dfs.disk.balancer.block.tolerance.percent is misleading
> --
>
> Key: HDFS-15440
> URL: https://issues.apache.org/jira/browse/HDFS-15440
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer mover
>Reporter: AMC-team
>Priority: Major
>
> In HDFS disk balancer, configuration parameter
> "dfs.disk.balancer.block.tolerance.percent" is to set a percentage (e.g. 10
> means 10%) which defines a good enough move.
> The description in hdfs-default.xml is not so clear to me how the value
> actually calculates and works
> {quote}When a disk balancer copy operation is proceeding, the datanode is
> still active. So it might not be possible to move the exactly specified
> amount of data. So tolerance allows us to define a percentage which defines a
> good enough move.
> {quote}
> So I refer to the [official doc of HDFS disk
> balancer|https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html]
> and the description is:
> {quote}The tolerance percent specifies when we have reached a good enough
> value for any copy step. For example, if you specify 10 then getting close to
> 10% of the target value is good enough. It is to say if the move operation is
> 20GB in size, if we can move 18GB (20 * (1-10%)) that operation is considered
> successful.
> {quote}
> However from the source code in DiskBalancer.java
> {code:java}
> // Inflates bytesCopied and returns true or false. This allows us to stop
> // copying if we have reached close enough.
> private boolean isCloseEnough(DiskBalancerWorkItem item) {
> long temp = item.getBytesCopied() +
> ((item.getBytesCopied() * getBlockTolerancePercentage(item)) / 100);
> return (item.getBytesToCopy() >= temp) ? false : true;
> }
> {code}
> Here, if item.getBytesToCopy() = 20GB, then item.getBytesCopied() = 18GB is
> still not enough because 20 > 18 + 18*0.1
> The calculation in isLessThanNeeded() (Checks if a given block is less than
> needed size to meet our goal.) is also not intuitive in the same way.
> *How to fix*
> Although this may not lead severe failure, it is better to make it consistent
> between doc and code, and also better to refine the description in
> hdfs-default.xml to make it more precise and clear.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15442) dfs.image.transfer.chunksize should have check code before use
[
https://issues.apache.org/jira/browse/HDFS-15442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
AMC-team updated HDFS-15442:
Description:
In current implementation of checkpoint image transfer, if the file length is
bigger than the configured value dfs.image.transfer.chunksize, it will use
chunked streaming mode to avoid internal buffering. This mode should be used
only if more than chunkSize data is present to upload, otherwise upload may not
happen sometimes.
{code:java}
//TransferFsImage.java
int chunkSize = (int) conf.getLongBytes(
DFSConfigKeys.DFS_IMAGE_TRANSFER_CHUNKSIZE_KEY,
DFSConfigKeys.DFS_IMAGE_TRANSFER_CHUNKSIZE_DEFAULT);
if (imageFile.length() > chunkSize) {
// using chunked streaming mode to support upload of 2GB+ files and to
// avoid internal buffering.
// this mode should be used only if more than chunkSize data is present
// to upload. otherwise upload may not happen sometimes.
connection.setChunkedStreamingMode(chunkSize);
}
{code}
There is no any check code for this parameter. User may accidentally set this
value to a wrong value. Here, if the user set chunkSize to a negative value.
Chunked streaming mode will always be used. In
setChunkedStreamingMode(chunkSize), there is a correction code that if the
chunkSize is <=0, it will be change to DEFAULT_CHUNK_SIZE.
{code:java}
public void setChunkedStreamingMode (int chunklen) {
if (connected) {
throw new IllegalStateException ("Can't set streaming mode: already
connected");
}
if (fixedContentLength != -1 || fixedContentLengthLong != -1) {
throw new IllegalStateException ("Fixed length streaming mode set");
}
chunkLength = chunklen <=0? DEFAULT_CHUNK_SIZE : chunklen;
}
{code}
However, the correction may be too late.
If the user set dfs.image.transfer.chunksize to value that <= 0, even for
images whose imageFile.length() < DEFAULT_CHUNK_SIZE will use chunked streaming
mode and may fail the upload as mentioned above. *(This scenario may not be
common, but* *we can prevent users setting this param to an extremely small
value.**)*
*How to fix:*
Add checking code or correction code right after parsing the config value
before really use the value (if statement and setChunkedStreamingMode).
was:
In current implementation of checkpoint image transfer, if the file length is
bigger than the configured value dfs.image.transfer.chunksize, it will use
chunked streaming mode to avoid internal buffering. This mode should be used
only if more than chunkSize data is present to upload, otherwise upload may not
happen sometimes.
{code:java}
//TransferFsImage.java
int chunkSize = (int) conf.getLongBytes(
DFSConfigKeys.DFS_IMAGE_TRANSFER_CHUNKSIZE_KEY,
DFSConfigKeys.DFS_IMAGE_TRANSFER_CHUNKSIZE_DEFAULT);
if (imageFile.length() > chunkSize) {
// using chunked streaming mode to support upload of 2GB+ files and to
// avoid internal buffering.
// this mode should be used only if more than chunkSize data is present
// to upload. otherwise upload may not happen sometimes.
connection.setChunkedStreamingMode(chunkSize);
}
{code}
There is no any check code for this parameter. User may accidentally set this
value to a wrong value. Here, if the user set chunkSize to a negative value.
Chunked streaming mode will always be used. In
setChunkedStreamingMode(chunkSize), there is a correction code that if the
chunkSize is <=0, it will be change to DEFAULT_CHUNK_SIZE.
{code:java}
public void setChunkedStreamingMode (int chunklen) {
if (connected) {
throw new IllegalStateException ("Can't set streaming mode: already
connected");
}
if (fixedContentLength != -1 || fixedContentLengthLong != -1) {
throw new IllegalStateException ("Fixed length streaming mode set");
}
chunkLength = chunklen <=0? DEFAULT_CHUNK_SIZE : chunklen;
}
{code}
However, the correction may be too late.
If the user set dfs.image.transfer.chunksize to value that <= 0, even for
images whose imageFile.length() < DEFAULT_CHUNK_SIZE will use chunked streaming
mode and may fail the upload as mentioned above. *(This scenario may not be
common, but a**t least we can prevent users setting this param to an extremely
small value.**)*
*How to fix:*
Add checking code or correction code right after parsing the config value
before really use the value (if statement and setChunkedStreamingMode).
> dfs.image.transfer.chunksize should have check code before use
> --
>
> Key: HDFS-15442
> URL: https://issues.apache.org/jira/browse/HDFS-15442
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: AMC-team
>Priority: Major
>
> In current implementation of checkpoint image transfer, if the file length is
> bigger than the configured value dfs.image.transfer.chunksize, it will use
[jira] [Updated] (HDFS-15439) Setting dfs.mover.retry.max.attempts to negative value will fail the retry policy.
[
https://issues.apache.org/jira/browse/HDFS-15439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
AMC-team updated HDFS-15439:
Summary: Setting dfs.mover.retry.max.attempts to negative value will fail
the retry policy. (was: dfs.mover.retry.max.attempts should not accept
negative value )
> Setting dfs.mover.retry.max.attempts to negative value will fail the retry
> policy.
> --
>
> Key: HDFS-15439
> URL: https://issues.apache.org/jira/browse/HDFS-15439
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer mover
>Reporter: AMC-team
>Priority: Major
>
> Configuration parameter "dfs.mover.retry.max.attempts" is to define the
> maximum number of retries before the mover consider the move failed. There is
> no checking code so this parameter can accept any int value.
> Theoratically, setting this value to <=0 should mean that no retry at all.
> However, if you set the value to negative value. The checking condition for
> retry failed will never satisfied because the if statement is "*if
> (retryCount.get() == retryMaxAttempts)*". The code is in Mover.java.
> {code:java}
> private Result processNamespace() throws IOException {
> ... //wait for pending move to finish and retry the failed migration
> if (hasFailed && !hasSuccess) {
> if (retryCount.get() == retryMaxAttempts) { // retryMaxAttempts < 0
> result.setRetryFailed();
> LOG.error("Failed to move some block's after "
> + retryMaxAttempts + " retries.");
> return result;
> } else {
> retryCount.incrementAndGet();
> }
> } else {
> // Reset retry count if no failure.
> retryCount.set(0);
> }
> ...
> }
> {code}
> *How to fix*
> Add checking code of "dfs.mover.retry.max.attempts" to accept only
> non-negative value or change the if statement condition when retry count
> exceeds max attempts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15421) IBR leak causes standby NN to be stuck in safe mode
[
https://issues.apache.org/jira/browse/HDFS-15421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-15421:
-
Target Version/s: 3.3.0, 3.2.2, 2.10.1, 3.1.5 (was: 3.3.0, 3.1.4, 3.2.2,
2.10.1)
> IBR leak causes standby NN to be stuck in safe mode
> ---
>
> Key: HDFS-15421
> URL: https://issues.apache.org/jira/browse/HDFS-15421
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Kihwal Lee
>Assignee: Akira Ajisaka
>Priority: Blocker
> Labels: release-blocker
> Fix For: 3.3.0, 3.2.2, 2.10.1, 3.1.5
>
> Attachments: HDFS-15421-000.patch, HDFS-15421-001.patch,
> HDFS-15421.002.patch, HDFS-15421.003.patch, HDFS-15421.004.patch,
> HDFS-15421.005.patch, HDFS-15421.006.patch, HDFS-15421.007.patch
>
>
> After HDFS-14941, update of the global gen stamp is delayed in certain
> situations. This makes the last set of incremental block reports from append
> "from future", which causes it to be simply re-queued to the pending DN
> message queue, rather than processed to complete the block. The last set of
> IBRs will leak and never cleaned until it transitions to active. The size of
> {{pendingDNMessages}} constantly grows until then.
> If a leak happens while in a startup safe mode, the namenode will never be
> able to come out of safe mode on its own.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14941) Potential editlog race condition can cause corrupted file
[
https://issues.apache.org/jira/browse/HDFS-14941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-14941:
-
Fix Version/s: (was: 3.1.4)
3.1.5
Reverted from branch-3.1.4. Updating the fix versions.
> Potential editlog race condition can cause corrupted file
> -
>
> Key: HDFS-14941
> URL: https://issues.apache.org/jira/browse/HDFS-14941
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Labels: ha
> Fix For: 3.3.0, 3.2.2, 2.10.1, 3.1.5
>
> Attachments: HDFS-14941.001.patch, HDFS-14941.002.patch,
> HDFS-14941.003.patch, HDFS-14941.004.patch, HDFS-14941.005.patch,
> HDFS-14941.006.patch
>
>
> Recently we encountered an issue that, after a failover, NameNode complains
> corrupted file/missing blocks. The blocks did recover after full block
> reports, so the blocks are not actually missing. After further investigation,
> we believe this is what happened:
> First of all, on SbN, it is possible that it receives block reports before
> corresponding edit tailing happened. In which case SbN postpones processing
> the DN block report, handled by the guarding logic below:
> {code:java}
> if (shouldPostponeBlocksFromFuture &&
> namesystem.isGenStampInFuture(iblk)) {
> queueReportedBlock(storageInfo, iblk, reportedState,
> QUEUE_REASON_FUTURE_GENSTAMP);
> continue;
> }
> {code}
> Basically if reported block has a future generation stamp, the DN report gets
> requeued.
> However, in {{FSNamesystem#storeAllocatedBlock}}, we have the following code:
> {code:java}
> // allocate new block, record block locations in INode.
> newBlock = createNewBlock();
> INodesInPath inodesInPath = INodesInPath.fromINode(pendingFile);
> saveAllocatedBlock(src, inodesInPath, newBlock, targets);
> persistNewBlock(src, pendingFile);
> offset = pendingFile.computeFileSize();
> {code}
> The line
> {{newBlock = createNewBlock();}}
> Would log an edit entry {{OP_SET_GENSTAMP_V2}} to bump generation stamp on
> Standby
> while the following line
> {{persistNewBlock(src, pendingFile);}}
> would log another edit entry {{OP_ADD_BLOCK}} to actually add the block on
> Standby.
> Then the race condition is that, imagine Standby has just processed
> {{OP_SET_GENSTAMP_V2}}, but not yet {{OP_ADD_BLOCK}} (if they just happen to
> be in different setment). Now a block report with new generation stamp comes
> in.
> Since the genstamp bump has already been processed, the reported block may
> not be considered as future block. So the guarding logic passes. But
> actually, the block hasn't been added to blockmap, because the second edit is
> yet to be tailed. So, the block then gets added to invalidate block list and
> we saw messages like:
> {code:java}
> BLOCK* addBlock: block XXX on node XXX size XXX does not belong to any file
> {code}
> Even worse, since this IBR is effectively lost, the NameNode has no
> information about this block, until the next full block report. So after a
> failover, the NN marks it as corrupt.
> This issue won't happen though, if both of the edit entries get tailed all
> together, so no IBR processing can happen in between. But in our case, we set
> edit tailing interval to super low (to allow Standby read), so when under
> high workload, there is a much much higher chance that the two entries are
> tailed separately, causing the issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15421) IBR leak causes standby NN to be stuck in safe mode
[
https://issues.apache.org/jira/browse/HDFS-15421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-15421:
-
Fix Version/s: 3.1.5
2.10.1
3.2.2
3.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed the 004 patch to trunk, branch-3.3, branch-3.3.0, branch-3.2,
branch-3.1, and branch-2.10. Thank you so much [~kihwal], [~shv],
[~vagarychen], and [~tasanuma] for reviewing.
> IBR leak causes standby NN to be stuck in safe mode
> ---
>
> Key: HDFS-15421
> URL: https://issues.apache.org/jira/browse/HDFS-15421
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Kihwal Lee
>Assignee: Akira Ajisaka
>Priority: Blocker
> Labels: release-blocker
> Fix For: 3.3.0, 3.2.2, 2.10.1, 3.1.5
>
> Attachments: HDFS-15421-000.patch, HDFS-15421-001.patch,
> HDFS-15421.002.patch, HDFS-15421.003.patch, HDFS-15421.004.patch,
> HDFS-15421.005.patch, HDFS-15421.006.patch, HDFS-15421.007.patch
>
>
> After HDFS-14941, update of the global gen stamp is delayed in certain
> situations. This makes the last set of incremental block reports from append
> "from future", which causes it to be simply re-queued to the pending DN
> message queue, rather than processed to complete the block. The last set of
> IBRs will leak and never cleaned until it transitions to active. The size of
> {{pendingDNMessages}} constantly grows until then.
> If a leak happens while in a startup safe mode, the namenode will never be
> able to come out of safe mode on its own.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15421) IBR leak causes standby NN to be stuck in safe mode
[
https://issues.apache.org/jira/browse/HDFS-15421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147233#comment-17147233
]
Hudson commented on HDFS-15421:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18387 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18387/])
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. (aajisaka: rev
c71ce7ac3370e220995bad0ae8b59d962c8d30a7)
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestUpdateBlockTailing.java
* (delete)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestAddBlockTailing.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirTruncateOp.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
> IBR leak causes standby NN to be stuck in safe mode
> ---
>
> Key: HDFS-15421
> URL: https://issues.apache.org/jira/browse/HDFS-15421
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Kihwal Lee
>Assignee: Akira Ajisaka
>Priority: Blocker
> Labels: release-blocker
> Attachments: HDFS-15421-000.patch, HDFS-15421-001.patch,
> HDFS-15421.002.patch, HDFS-15421.003.patch, HDFS-15421.004.patch,
> HDFS-15421.005.patch, HDFS-15421.006.patch, HDFS-15421.007.patch
>
>
> After HDFS-14941, update of the global gen stamp is delayed in certain
> situations. This makes the last set of incremental block reports from append
> "from future", which causes it to be simply re-queued to the pending DN
> message queue, rather than processed to complete the block. The last set of
> IBRs will leak and never cleaned until it transitions to active. The size of
> {{pendingDNMessages}} constantly grows until then.
> If a leak happens while in a startup safe mode, the namenode will never be
> able to come out of safe mode on its own.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
