[jira] [Work logged] (HDFS-15600) TestRouterQuota fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-15600?focusedWorklogId=492255=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-492255
]
ASF GitHub Bot logged work on HDFS-15600:
-
Author: ASF GitHub Bot
Created on: 29/Sep/20 03:41
Start Date: 29/Sep/20 03:41
Worklog Time Spent: 10m
Work Description: liuml07 commented on pull request #2347:
URL: https://github.com/apache/hadoop/pull/2347#issuecomment-700407063
I have not had a second look at the original patch regarding compatibility,
but we can merge this patch first to make test in trunk pass.
If we need to make any further changes to the original NVDIMM feature eg
reordering, we can also update this test again.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 492255)
Time Spent: 20m (was: 10m)
> TestRouterQuota fails in trunk
> --
>
> Key: HDFS-15600
> URL: https://issues.apache.org/jira/browse/HDFS-15600
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The test is failing due to addition of a new storage type {{NVDIMM}} in
> middle.
> Ref :
> https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/204/testReport/org.apache.hadoop.hdfs.server.federation.router/TestRouterQuota/testStorageTypeQuota/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203594#comment-17203594 ] huangtianhua commented on HDFS-15025: - And I have proposed a PR to fix the test "TestRouterQuota" https://github.com/apache/hadoop/pull/2347 > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 12h 10m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15600) TestRouterQuota fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-15600?focusedWorklogId=49=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-49
]
ASF GitHub Bot logged work on HDFS-15600:
-
Author: ASF GitHub Bot
Created on: 29/Sep/20 01:56
Start Date: 29/Sep/20 01:56
Worklog Time Spent: 10m
Work Description: huangtianhua opened a new pull request #2347:
URL: https://github.com/apache/hadoop/pull/2347
After "NVDIMM" adding, we have six storage types. This
fixes the related test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 49)
Remaining Estimate: 0h
Time Spent: 10m
> TestRouterQuota fails in trunk
> --
>
> Key: HDFS-15600
> URL: https://issues.apache.org/jira/browse/HDFS-15600
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Ayush Saxena
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The test is failing due to addition of a new storage type {{NVDIMM}} in
> middle.
> Ref :
> https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/204/testReport/org.apache.hadoop.hdfs.server.federation.router/TestRouterQuota/testStorageTypeQuota/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15600) TestRouterQuota fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-15600:
--
Labels: pull-request-available (was: )
> TestRouterQuota fails in trunk
> --
>
> Key: HDFS-15600
> URL: https://issues.apache.org/jira/browse/HDFS-15600
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The test is failing due to addition of a new storage type {{NVDIMM}} in
> middle.
> Ref :
> https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/204/testReport/org.apache.hadoop.hdfs.server.federation.router/TestRouterQuota/testStorageTypeQuota/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15600) TestRouterQuota fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203580#comment-17203580
]
huangtianhua commented on HDFS-15600:
-
[~elgoiri] OK, I will propose a pr to fix this.
> TestRouterQuota fails in trunk
> --
>
> Key: HDFS-15600
> URL: https://issues.apache.org/jira/browse/HDFS-15600
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Ayush Saxena
>Priority: Major
>
> The test is failing due to addition of a new storage type {{NVDIMM}} in
> middle.
> Ref :
> https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/204/testReport/org.apache.hadoop.hdfs.server.federation.router/TestRouterQuota/testStorageTypeQuota/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203240#comment-17203240
]
YaYun Wang edited comment on HDFS-15025 at 9/29/20, 1:13 AM:
-
[~liuml07]，[~ayushtkn]， we have tried to make sure that {{NVDIMM}} and
{{isRAM}} will not affect the function and existing data on hadoop by
flollowing ways:
* *Running test case*: run test cases on
{{FsEditLogOp#SetQuotaByStorageTypeOp}}, such as {{TestWebHDFS}}, and that can
pass without failures or errors.
* *Combing the classes about {{FsEditLogOp＃SetQuotaByStorageTypeOp}}*,
including {{DistributedFileSystem}}, {{DFSClient}}, {{NamenodeRPCServer}},
{{FSNamesystem}}, {{FSDirAttrOp}}, {{FSEditLog}} and {{FSEditLogOp,}} etc.. We
can draw a conclusion that {{ordinal()}} of StorageType which is enum is
flexible and compatible, that is, {{ordinal()}} can vary with the value of
{{StorageType}} and that's different with
{{TestRouterQuota.testStorageTypeQuota}} where {{ordinal()}} is used as the
index of array.
{code:java}
public void testStorageTypeQuota() throws Exception {
...
//the first parameters is a array which length is 5 for 5 storage types.
verifyTypeQuotaAndConsume(new long[] {-1, -1, ssQuota * 2, -1, -1}, null,
usage);
...
}
private void verifyTypeQuotaAndConsume(long[] quota, long[] consume,
QuotaUsage usage) {
for (StorageType t : StorageType.values()) {
if (quota != null) {
assertEquals(quota[t.ordinal()], usage.getTypeQuota(t)); //ordinal() is
the index of quota
}
if (consume != null) {
assertEquals(consume[t.ordinal()], usage.getTypeConsumed(t));
}
}
}{code}
* *Verifing the function of hadoop with {{NVDIMM}} and {{isRAM}}* : the
functions of DFSIO, wordcount, put/get built-in hadoop can used normally. The
old storage types and storage policies are normal too after upgrade.
* *Verifing the function after upgrade by existing data*: first, we write data
to hadoop without the patch using DFSIO, wordcount and put/get. Then update the
version of hadoop with the patch. After that, the existing data of old version
can be accessed and used normally.
was (Author: wangyayun):
[~liuml07]，[~ayushtkn]， we have tried to make sure that {{NVDIMM}} and
{{isRAM}} will not affect the function and existing data on hadoop by
flollowing ways:
# Running test case: run test cases on
{{FsEditLogOp#SetQuotaByStorageTypeOp}}, such as {{TestWebHDFS}}, and that can
pass without failures or errors.
# Combing the code about {{FsEditLogOp＃SetQuotaByStorageTypeOp}}, including
classes of {{DistributedFileSystem}}, {{DFSClient}}, {{NamenodeRPCServer}},
{{FSNamesystem}}, {{FSDirAttrOp}}, {{FSEditLog}} and {{FSEditLogOp,}} etc.. We
can draw a conclusion that {{ordinal()}} of StorageType which is enum is
flexible and compatible, that is, {{ordinal()}} can vary with the value of
{{StorageType}} and that's different with
{{TestRouterQuota.testStorageTypeQuota}} where {{ordinal()}} is used as the
index of array.
{code:java}
public void testStorageTypeQuota() throws Exception {
...
//the first parameters is a array which length is 5 for 5 storage types.
verifyTypeQuotaAndConsume(new long[] {-1, -1, ssQuota * 2, -1, -1}, null,
usage);
...
}
private void verifyTypeQuotaAndConsume(long[] quota, long[] consume,
QuotaUsage usage) {
for (StorageType t : StorageType.values()) {
if (quota != null) {
assertEquals(quota[t.ordinal()], usage.getTypeQuota(t)); //ordinal() is
the index of quota
}
if (consume != null) {
assertEquals(consume[t.ordinal()], usage.getTypeConsumed(t));
}
}
}{code}
# Verifing the function of hadoop with {{NVDIMM}} and {{isRAM}} : the
functions of DFSIO, wordcount, put/get built-in hadoop can used normally. The
old storage types and storage policies are normal too after upgrade.
# Verifing the function after upgrade by existing data: first, we write data
to hadoop without the patch using DFSIO, wordcount and put/get. Then update the
version of hadoop with the patch. After that, the existing data of old version
can be accessed and used normally.
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 12h 10m
> Remaining Estimate: 0h
>
> The non-volatile
[jira] [Comment Edited] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203240#comment-17203240
]
YaYun Wang edited comment on HDFS-15025 at 9/29/20, 1:08 AM:
-
[~liuml07]，[~ayushtkn]， we have tried to make sure that {{NVDIMM}} and
{{isRAM}} will not affect the function and existing data on hadoop by
flollowing ways:
# Running test case: run test cases on
{{FsEditLogOp#SetQuotaByStorageTypeOp}}, such as {{TestWebHDFS}}, and that can
pass without failures or errors.
# Combing the code about {{FsEditLogOp＃SetQuotaByStorageTypeOp}}, including
classes of {{DistributedFileSystem}}, {{DFSClient}}, {{NamenodeRPCServer}},
{{FSNamesystem}}, {{FSDirAttrOp}}, {{FSEditLog}} and {{FSEditLogOp,}} etc.. We
can draw a conclusion that {{ordinal()}} of StorageType which is enum is
flexible and compatible, that is, {{ordinal()}} can vary with the value of
{{StorageType}} and that's different with
{{TestRouterQuota.testStorageTypeQuota}} where {{ordinal()}} is used as the
index of array.
{code:java}
public void testStorageTypeQuota() throws Exception {
...
//the first parameters is a array which length is 5 for 5 storage types.
verifyTypeQuotaAndConsume(new long[] {-1, -1, ssQuota * 2, -1, -1}, null,
usage);
...
}
private void verifyTypeQuotaAndConsume(long[] quota, long[] consume,
QuotaUsage usage) {
for (StorageType t : StorageType.values()) {
if (quota != null) {
assertEquals(quota[t.ordinal()], usage.getTypeQuota(t)); //ordinal() is
the index of quota
}
if (consume != null) {
assertEquals(consume[t.ordinal()], usage.getTypeConsumed(t));
}
}
}{code}
# Verifing the function of hadoop with {{NVDIMM}} and {{isRAM}} : the
functions of DFSIO, wordcount, put/get built-in hadoop can used normally. The
old storage types and storage policies are normal too after upgrade.
# Verifing the function after upgrade by existing data: first, we write data
to hadoop without the patch using DFSIO, wordcount and put/get. Then update the
version of hadoop with the patch. After that, the existing data of old version
can be accessed and used normally.
was (Author: wangyayun):
[~liuml07]，[~ayushtkn]， we have tried to make sure that {{NVDIMM}} and
{{isRAM}} will not affect the function and existing data on hadoop by
flollowing ways:
# Running test case: run test cases on
{{FsEditLogOp#SetQuotaByStorageTypeOp}}, such as {{TestWebHDFS}}, and that can
pass without failures or errors.
# Combing the code about {{FsEditLogOp＃SetQuotaByStorageTypeOp}}, including
{{DistributedFileSystem}}, {{DFSClient}}, {{NamenodeRPCServer}},
{{FSNamesystem}}, {{FSDirAttrOp}}, {{FSEditLog}} and {{FSEditLogOp,}} etc.. We
can draw a conclusion that {{ordinal()}} of StorageType which is enum is
flexible and compatible, that is, {{ordinal()}} can vary with the value of
{{StorageType}} and that's different with
{{TestRouterQuota.testStorageTypeQuota}} where {{ordinal()}} is used as the
index of array.
{code:java}
public void testStorageTypeQuota() throws Exception {
...
//the first parameters is a array which length is 5 for 5 storage types.
verifyTypeQuotaAndConsume(new long[] {-1, -1, ssQuota * 2, -1, -1}, null,
usage);
...
}
private void verifyTypeQuotaAndConsume(long[] quota, long[] consume,
QuotaUsage usage) {
for (StorageType t : StorageType.values()) {
if (quota != null) {
assertEquals(quota[t.ordinal()], usage.getTypeQuota(t)); //ordinal() is
the index of quota
}
if (consume != null) {
assertEquals(consume[t.ordinal()], usage.getTypeConsumed(t));
}
}
}{code}
# Verifing the function of hadoop with {{NVDIMM}} and {{isRAM}} : the
functions of DFSIO, wordcount, put/get built-in hadoop can used normally. The
old storage types and storage policies are normal too after upgrade.
# Verifing the function after upgrade by existing data: first, we write data
to hadoop without the patch using DFSIO, wordcount and put/get. Then updata the
version of hadoop with the patch. After that, the existing data of old version
can be accessed and used normally.
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 12h 10m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is
[jira] [Comment Edited] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203240#comment-17203240
]
YaYun Wang edited comment on HDFS-15025 at 9/29/20, 1:04 AM:
-
[~liuml07]，[~ayushtkn]， we have tried to make sure that {{NVDIMM}} and
{{isRAM}} will not affect the function and existing data on hadoop by
flollowing ways:
# Running test case: run test cases on
{{FsEditLogOp#SetQuotaByStorageTypeOp}}, such as {{TestWebHDFS}}, and that can
pass without failures or errors.
# Combing the code about {{FsEditLogOp＃SetQuotaByStorageTypeOp}}, including
{{DistributedFileSystem}}, {{DFSClient}}, {{NamenodeRPCServer}},
{{FSNamesystem}}, {{FSDirAttrOp}}, {{FSEditLog}} and {{FSEditLogOp,}} etc.. We
can draw a conclusion that {{ordinal()}} of StorageType which is enum is
flexible and compatible, that is, {{ordinal()}} can vary with the value of
{{StorageType}} and that's different with
{{TestRouterQuota.testStorageTypeQuota}} where {{ordinal()}} is used as the
index of array.
{code:java}
public void testStorageTypeQuota() throws Exception {
...
//the first parameters is a array which length is 5 for 5 storage types.
verifyTypeQuotaAndConsume(new long[] {-1, -1, ssQuota * 2, -1, -1}, null,
usage);
...
}
private void verifyTypeQuotaAndConsume(long[] quota, long[] consume,
QuotaUsage usage) {
for (StorageType t : StorageType.values()) {
if (quota != null) {
assertEquals(quota[t.ordinal()], usage.getTypeQuota(t)); //ordinal() is
the index of quota
}
if (consume != null) {
assertEquals(consume[t.ordinal()], usage.getTypeConsumed(t));
}
}
}{code}
# Verifing the function of hadoop with {{NVDIMM}} and {{isRAM}} : the
functions of DFSIO, wordcount, put/get built-in hadoop can used normally. The
old storage types and storage policies are normal too after upgrade.
# Verifing the function after upgrade by existing data: first, we write data
to hadoop without the patch using DFSIO, wordcount and put/get. Then updata the
version of hadoop with the patch. After that, the existing data of old version
can be accessed and used normally.
was (Author: wangyayun):
[~liuml07]，[~ayushtkn]， we have tried to make sure that {{NVDIMM}} and
{{isRAM}} will not affect the function and existing data on hadoop by
flollowing ways:
# Running test case: run test cases on
{{FsEditLogOp#SetQuotaByStorageTypeOp}}, such as {{TestWebHDFS}}, and that can
pass without failures or errors.
# Combing the code about {{FsEditLogOp＃SetQuotaByStorageTypeOp}}, including
{{DistributedFileSystem}}, {{DFSClient}}, {{NamenodeRPCServer}},
{{FSNamesystem}}, {{FSDirAttrOp}}, {{FSEditLog}} and {{FSEditLogOp,}} etc.. We
can draw a conclusion that {{ordinal()}} of StorageType which is enum is
flexible, that is, {{ordinal()}} can vary with the value of {{StorageType}} and
that's different with {{TestRouterQuota.testStorageTypeQuota}} where
{{ordinal()}} is used as the index of array.
# Verifing the function of hadoop with {{NVDIMM}} and {{isRAM}} : the
functions of DFSIO, wordcount, put/get built-in hadoop can used normally. The
old storage types and storage policies are normal too.
# Verifing the function after upgrade by existing data: in the first, we write
data to hadoop without the patch using DFSIO, wordcount and put/get, then
updata the version of hadoop with the patch, and the existing data of old
version can be accessed and used normally after upgrade.
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 12h 10m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15415) Reduce locking in Datanode DirectoryScanner
[
https://issues.apache.org/jira/browse/HDFS-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203565#comment-17203565
]
Hadoop QA commented on HDFS-15415:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
44s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
59s{color} | {color:green}{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green}{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green}{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 15s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} branch-3.1 passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
28s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
27s{color} | {color:green}{color} | {color:green} branch-3.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 45s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/212/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt{color}
| {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new +
39 unchanged - 2 fixed = 41 total (was 41) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 45s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green}{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}138m 10s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/212/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
59s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/212/artifact/out/patch-asflicense-problems.txt{color}
| {color:red} The patch generated 18 ASF License warnings.
[jira] [Commented] (HDFS-15415) Reduce locking in Datanode DirectoryScanner
[
https://issues.apache.org/jira/browse/HDFS-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203511#comment-17203511
]
Stephen O'Donnell commented on HDFS-15415:
--
Updated a patch for branch-3.1. With this patch on 3.1, the diff between it and
the commited patch of 3.2 is:
{code}
$ git diff backport-HDFS-15415-3.1..origin/branch-3.2
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
@@ -37,9 +37,9 @@
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
-import org.apache.commons.lang.time.FastDateFormat;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
+import org.apache.commons.lang3.time.FastDateFormat;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.StorageType;
@@ -58,7 +58,8 @@
*/
@InterfaceAudience.Private
public class DirectoryScanner implements Runnable {
- private static final Log LOG = LogFactory.getLog(DirectoryScanner.class);
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DirectoryScanner.class);
private static final int MILLIS_PER_SECOND = 1000;
private static final String START_MESSAGE =
"Periodic Directory Tree Verification scan"
@@ -163,7 +164,7 @@ public String toString() {
/**
* Merges {@code that} ScanInfoPerBlockPool into this one
*
- * @param the ScanInfoPerBlockPool to merge
+ * @param that ScanInfoPerBlockPool to merge
*/
public void addAll(ScanInfoPerBlockPool that) {
if (that == null) return;
@@ -459,9 +460,9 @@ private void scan() {
d++;
if (d < blockpoolReport.length) {
- // There may be multiple on-disk records for the same block, don't
increment
- // the memory record pointer if so.
- ScanInfo nextInfo = blockpoolReport[Math.min(d,
blockpoolReport.length - 1)];
+ // There may be multiple on-disk records for the same block,
+ // don't increment the memory record pointer if so.
+ ScanInfo nextInfo = blockpoolReport[d];
if (nextInfo.getBlockId() != info.getBlockId()) {
++m;
}
{code}
This patch only changes the scan method, and note the only difference between
the two files is the change added by HDFS-13829 on branch-3.2.
> Reduce locking in Datanode DirectoryScanner
> ---
>
> Key: HDFS-15415
> URL: https://issues.apache.org/jira/browse/HDFS-15415
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.4.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
> Attachments: HDFS-15415.001.patch, HDFS-15415.002.patch,
> HDFS-15415.003.patch, HDFS-15415.004.patch, HDFS-15415.005.patch,
> HDFS-15415.branch-3.1.001.patch, HDFS-15415.branch-3.2.001.patch,
> HDFS-15415.branch-3.2.002.patch, HDFS-15415.branch-3.3.001.patch
>
>
> In HDFS-15406, we have a small change to greatly reduce the runtime and
> locking time of the datanode DirectoryScanner. They may be room for further
> improvement.
> From the scan step, we have captured a snapshot of what is on disk. After
> calling `dataset.getFinalizedBlocks(bpid);` we have taken a snapshot of in
> memory. The two snapshots are never 100% in sync as things are always
> changing as the disk is scanned.
> We are only comparing finalized blocks, so they should not really change:
> * If a block is deleted after our snapshot, our snapshot will not see it and
> that is OK.
> * A finalized block could be appended. If that happens both the genstamp and
> length will change, but that should be handled by reconcile when it calls
> `FSDatasetImpl.checkAndUpdate()`, and there is nothing stopping blocks being
> appended after they have been scanned from disk, but before they have been
> compared with memory.
> My suspicion is that we can do all the comparison work outside of the lock
> and checkAndUpdate() re-checks any differences later under the lock on a
> block by block basis.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15415) Reduce locking in Datanode DirectoryScanner
[ https://issues.apache.org/jira/browse/HDFS-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDFS-15415: - Attachment: HDFS-15415.branch-3.1.001.patch > Reduce locking in Datanode DirectoryScanner > --- > > Key: HDFS-15415 > URL: https://issues.apache.org/jira/browse/HDFS-15415 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Fix For: 3.3.1, 3.4.0 > > Attachments: HDFS-15415.001.patch, HDFS-15415.002.patch, > HDFS-15415.003.patch, HDFS-15415.004.patch, HDFS-15415.005.patch, > HDFS-15415.branch-3.1.001.patch, HDFS-15415.branch-3.2.001.patch, > HDFS-15415.branch-3.2.002.patch, HDFS-15415.branch-3.3.001.patch > > > In HDFS-15406, we have a small change to greatly reduce the runtime and > locking time of the datanode DirectoryScanner. They may be room for further > improvement. > From the scan step, we have captured a snapshot of what is on disk. After > calling `dataset.getFinalizedBlocks(bpid);` we have taken a snapshot of in > memory. The two snapshots are never 100% in sync as things are always > changing as the disk is scanned. > We are only comparing finalized blocks, so they should not really change: > * If a block is deleted after our snapshot, our snapshot will not see it and > that is OK. > * A finalized block could be appended. If that happens both the genstamp and > length will change, but that should be handled by reconcile when it calls > `FSDatasetImpl.checkAndUpdate()`, and there is nothing stopping blocks being > appended after they have been scanned from disk, but before they have been > compared with memory. > My suspicion is that we can do all the comparison work outside of the lock > and checkAndUpdate() re-checks any differences later under the lock on a > block by block basis. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15415) Reduce locking in Datanode DirectoryScanner
[ https://issues.apache.org/jira/browse/HDFS-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203498#comment-17203498 ] Stephen O'Donnell commented on HDFS-15415: -- [~weichiu] Thanks for committing. There is conflict getting this back to 3.1 due to HDFS-13829 missing on branch-3.1. Let me see if I can create one final branch-3.1 patch. > Reduce locking in Datanode DirectoryScanner > --- > > Key: HDFS-15415 > URL: https://issues.apache.org/jira/browse/HDFS-15415 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Fix For: 3.3.1, 3.4.0 > > Attachments: HDFS-15415.001.patch, HDFS-15415.002.patch, > HDFS-15415.003.patch, HDFS-15415.004.patch, HDFS-15415.005.patch, > HDFS-15415.branch-3.2.001.patch, HDFS-15415.branch-3.2.002.patch, > HDFS-15415.branch-3.3.001.patch > > > In HDFS-15406, we have a small change to greatly reduce the runtime and > locking time of the datanode DirectoryScanner. They may be room for further > improvement. > From the scan step, we have captured a snapshot of what is on disk. After > calling `dataset.getFinalizedBlocks(bpid);` we have taken a snapshot of in > memory. The two snapshots are never 100% in sync as things are always > changing as the disk is scanned. > We are only comparing finalized blocks, so they should not really change: > * If a block is deleted after our snapshot, our snapshot will not see it and > that is OK. > * A finalized block could be appended. If that happens both the genstamp and > length will change, but that should be handled by reconcile when it calls > `FSDatasetImpl.checkAndUpdate()`, and there is nothing stopping blocks being > appended after they have been scanned from disk, but before they have been > compared with memory. > My suspicion is that we can do all the comparison work outside of the lock > and checkAndUpdate() re-checks any differences later under the lock on a > block by block basis. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12861) Track speed in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-12861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203489#comment-17203489
]
Stephen O'Donnell commented on HDFS-12861:
--
It would be great to move this change forward along with HDFS-14084. Its a
fairly big patch though. I have not looked at it in detail, but I wonder if we
could break it down into a few smaller patches?
> Track speed in DFSClient
>
>
> Key: HDFS-12861
> URL: https://issues.apache.org/jira/browse/HDFS-12861
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: María Fernanda Borge
>Priority: Major
> Attachments: HDFS-12861-10-april-18.patch
>
>
> Sometimes we get slow jobs because of the access to HDFS. However, is hard to
> tell what is the actual speed. We propose to add a log line with something
> like:
> {code}
> 2017-11-19 09:55:26,309 INFO [main] hdfs.DFSClient: blk_1107222019_38144502
> READ 129500B in 7ms 17.6MB/s
> 2017-11-27 19:01:04,141 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792057_86833357 WRITE 131072B in 10ms 12.5MB/s
> 2017-11-27 19:01:14,219 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792069_86833369 WRITE 131072B in 12ms 10.4MB/s
> 2017-11-27 19:01:24,282 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792081_86833381 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:34,330 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792093_86833393 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:44,408 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792105_86833405 WRITE 131072B in 11ms 11.4MB/s
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14084) Need for more stats in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203485#comment-17203485 ] Erik Krogen commented on HDFS-14084: Hey [~sodonnell], thanks for the ping. Internally we decided to go with a different approach so I never got around to wrapping this up. Unfortunately I'm not working on HDFS these days and don't have the bandwidth to devote to it beyond maybe some simple reviews. I've unassigned this ticket from myself. > Need for more stats in DFSClient > > > Key: HDFS-14084 > URL: https://issues.apache.org/jira/browse/HDFS-14084 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Pranay Singh >Priority: Minor > Attachments: HDFS-14084.001.patch, HDFS-14084.002.patch, > HDFS-14084.003.patch, HDFS-14084.004.patch, HDFS-14084.005.patch, > HDFS-14084.006.patch, HDFS-14084.007.patch, HDFS-14084.008.patch, > HDFS-14084.009.patch, HDFS-14084.010.patch, HDFS-14084.011.patch, > HDFS-14084.012.patch, HDFS-14084.013.patch, HDFS-14084.014.patch, > HDFS-14084.015.patch, HDFS-14084.016.patch, HDFS-14084.017.patch, > HDFS-14084.018.patch > > > The usage of HDFS has changed from being used as a map-reduce filesystem, now > it's becoming more of like a general purpose filesystem. In most of the cases > there are issues with the Namenode so we have metrics to know the workload or > stress on Namenode. > However, there is a need to have more statistics collected for different > operations/RPCs in DFSClient to know which RPC operations are taking longer > time or to know what is the frequency of the operation.These statistics can > be exposed to the users of DFS Client and they can periodically log or do > some sort of flow control if the response is slow. This will also help to > isolate HDFS issue in a mixed environment where on a node say we have Spark, > HBase and Impala running together. We can check the throughput of different > operation across client and isolate the problem caused because of noisy > neighbor or network congestion or shared JVM. > We have dealt with several problems from the field for which there is no > conclusive evidence as to what caused the problem. If we had metrics or stats > in DFSClient we would be better equipped to solve such complex problems. > List of jiras for reference: > - > HADOOP-15538 HADOOP-15530 ( client side deadlock) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14084) Need for more stats in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen reassigned HDFS-14084: -- Assignee: (was: Erik Krogen) > Need for more stats in DFSClient > > > Key: HDFS-14084 > URL: https://issues.apache.org/jira/browse/HDFS-14084 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Pranay Singh >Priority: Minor > Attachments: HDFS-14084.001.patch, HDFS-14084.002.patch, > HDFS-14084.003.patch, HDFS-14084.004.patch, HDFS-14084.005.patch, > HDFS-14084.006.patch, HDFS-14084.007.patch, HDFS-14084.008.patch, > HDFS-14084.009.patch, HDFS-14084.010.patch, HDFS-14084.011.patch, > HDFS-14084.012.patch, HDFS-14084.013.patch, HDFS-14084.014.patch, > HDFS-14084.015.patch, HDFS-14084.016.patch, HDFS-14084.017.patch, > HDFS-14084.018.patch > > > The usage of HDFS has changed from being used as a map-reduce filesystem, now > it's becoming more of like a general purpose filesystem. In most of the cases > there are issues with the Namenode so we have metrics to know the workload or > stress on Namenode. > However, there is a need to have more statistics collected for different > operations/RPCs in DFSClient to know which RPC operations are taking longer > time or to know what is the frequency of the operation.These statistics can > be exposed to the users of DFS Client and they can periodically log or do > some sort of flow control if the response is slow. This will also help to > isolate HDFS issue in a mixed environment where on a node say we have Spark, > HBase and Impala running together. We can check the throughput of different > operation across client and isolate the problem caused because of noisy > neighbor or network congestion or shared JVM. > We have dealt with several problems from the field for which there is no > conclusive evidence as to what caused the problem. If we had metrics or stats > in DFSClient we would be better equipped to solve such complex problems. > List of jiras for reference: > - > HADOOP-15538 HADOOP-15530 ( client side deadlock) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15415) Reduce locking in Datanode DirectoryScanner
[ https://issues.apache.org/jira/browse/HDFS-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203483#comment-17203483 ] Wei-Chiu Chuang commented on HDFS-15415: Ok +1. Committed the patch into branch-3.2 > Reduce locking in Datanode DirectoryScanner > --- > > Key: HDFS-15415 > URL: https://issues.apache.org/jira/browse/HDFS-15415 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Fix For: 3.3.1, 3.4.0 > > Attachments: HDFS-15415.001.patch, HDFS-15415.002.patch, > HDFS-15415.003.patch, HDFS-15415.004.patch, HDFS-15415.005.patch, > HDFS-15415.branch-3.2.001.patch, HDFS-15415.branch-3.2.002.patch, > HDFS-15415.branch-3.3.001.patch > > > In HDFS-15406, we have a small change to greatly reduce the runtime and > locking time of the datanode DirectoryScanner. They may be room for further > improvement. > From the scan step, we have captured a snapshot of what is on disk. After > calling `dataset.getFinalizedBlocks(bpid);` we have taken a snapshot of in > memory. The two snapshots are never 100% in sync as things are always > changing as the disk is scanned. > We are only comparing finalized blocks, so they should not really change: > * If a block is deleted after our snapshot, our snapshot will not see it and > that is OK. > * A finalized block could be appended. If that happens both the genstamp and > length will change, but that should be handled by reconcile when it calls > `FSDatasetImpl.checkAndUpdate()`, and there is nothing stopping blocks being > appended after they have been scanned from disk, but before they have been > compared with memory. > My suspicion is that we can do all the comparison work outside of the lock > and checkAndUpdate() re-checks any differences later under the lock on a > block by block basis. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14084) Need for more stats in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203481#comment-17203481 ] Stephen O'Donnell commented on HDFS-14084: -- [~xkrogen] It looks like this Jira was very nearly done, and it is something that would be great to get committed. Do you have bandwidth to finish it off? > Need for more stats in DFSClient > > > Key: HDFS-14084 > URL: https://issues.apache.org/jira/browse/HDFS-14084 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Pranay Singh >Assignee: Erik Krogen >Priority: Minor > Attachments: HDFS-14084.001.patch, HDFS-14084.002.patch, > HDFS-14084.003.patch, HDFS-14084.004.patch, HDFS-14084.005.patch, > HDFS-14084.006.patch, HDFS-14084.007.patch, HDFS-14084.008.patch, > HDFS-14084.009.patch, HDFS-14084.010.patch, HDFS-14084.011.patch, > HDFS-14084.012.patch, HDFS-14084.013.patch, HDFS-14084.014.patch, > HDFS-14084.015.patch, HDFS-14084.016.patch, HDFS-14084.017.patch, > HDFS-14084.018.patch > > > The usage of HDFS has changed from being used as a map-reduce filesystem, now > it's becoming more of like a general purpose filesystem. In most of the cases > there are issues with the Namenode so we have metrics to know the workload or > stress on Namenode. > However, there is a need to have more statistics collected for different > operations/RPCs in DFSClient to know which RPC operations are taking longer > time or to know what is the frequency of the operation.These statistics can > be exposed to the users of DFS Client and they can periodically log or do > some sort of flow control if the response is slow. This will also help to > isolate HDFS issue in a mixed environment where on a node say we have Spark, > HBase and Impala running together. We can check the throughput of different > operation across client and isolate the problem caused because of noisy > neighbor or network congestion or shared JVM. > We have dealt with several problems from the field for which there is no > conclusive evidence as to what caused the problem. If we had metrics or stats > in DFSClient we would be better equipped to solve such complex problems. > List of jiras for reference: > - > HADOOP-15538 HADOOP-15530 ( client side deadlock) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15603) RBF: Fix getLocationsForPath twice in create operation
[ https://issues.apache.org/jira/browse/HDFS-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15603: Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > RBF: Fix getLocationsForPath twice in create operation > -- > > Key: HDFS-15603 > URL: https://issues.apache.org/jira/browse/HDFS-15603 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15603-001.patch > > > getLocationsForPath in create(), but getLocationsForPath again in > getCreateLocation(),is not necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15603) RBF: Fix getLocationsForPath twice in create operation
[ https://issues.apache.org/jira/browse/HDFS-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203455#comment-17203455 ] Ayush Saxena commented on HDFS-15603: - Committed to trunk. Thanx [~wangzhaohui] for the contribution, [~elgoiri] and [~hexiaoqiao] for the reviews!!! > RBF: Fix getLocationsForPath twice in create operation > -- > > Key: HDFS-15603 > URL: https://issues.apache.org/jira/browse/HDFS-15603 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > Attachments: HDFS-15603-001.patch > > > getLocationsForPath in create(), but getLocationsForPath again in > getCreateLocation(),is not necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15591) RBF: Fix webHdfs file display error
[ https://issues.apache.org/jira/browse/HDFS-15591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15591: Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > RBF: Fix webHdfs file display error > --- > > Key: HDFS-15591 > URL: https://issues.apache.org/jira/browse/HDFS-15591 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15591-001.patch, HDFS-15591-002.patch, > HDFS-15591-003.patch, HDFS-15591-004.patch, RBF_Browse_Directory.png, > RBF_Browse_Directory_PostFix.png, after-1.jpg, after-2.jpg, before-1.jpg, > before-2.jpg > > > The path mounted by the router does not exist on NN，router will create > virtual folder with the mount name, but the "browse the file syaytem" display > on http is wrong. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15591) RBF: Fix webHdfs file display error
[ https://issues.apache.org/jira/browse/HDFS-15591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203449#comment-17203449 ] Ayush Saxena commented on HDFS-15591: - Committed to trunk. Thanx [~wangzhaohui] for the contribution and [~elgoiri] for the review!!! > RBF: Fix webHdfs file display error > --- > > Key: HDFS-15591 > URL: https://issues.apache.org/jira/browse/HDFS-15591 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > Attachments: HDFS-15591-001.patch, HDFS-15591-002.patch, > HDFS-15591-003.patch, HDFS-15591-004.patch, RBF_Browse_Directory.png, > RBF_Browse_Directory_PostFix.png, after-1.jpg, after-2.jpg, before-1.jpg, > before-2.jpg > > > The path mounted by the router does not exist on NN，router will create > virtual folder with the mount name, but the "browse the file syaytem" display > on http is wrong. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15603) RBF: Fix getLocationsForPath twice in create operation
[ https://issues.apache.org/jira/browse/HDFS-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203446#comment-17203446 ] Ayush Saxena commented on HDFS-15603: - +1, The existing tests are enough to check this. This doesn't change any functionality. > RBF: Fix getLocationsForPath twice in create operation > -- > > Key: HDFS-15603 > URL: https://issues.apache.org/jira/browse/HDFS-15603 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > Attachments: HDFS-15603-001.patch > > > getLocationsForPath in create(), but getLocationsForPath again in > getCreateLocation(),is not necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14553) Make queue size of BlockReportProcessingThread configurable
[ https://issues.apache.org/jira/browse/HDFS-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203414#comment-17203414 ] Xiaoyu Yao commented on HDFS-14553: --- Thanks [~hexiaoqiao] for the information. Do you change the threshold for separate block report feature from HDFS-5153? How big is your cluster when the queue size is 4096? I've seen issues when the queue is full, lots of server handler threads got stucked on it with the default size of 1024 for a cluster of size about 1K nodes. > Make queue size of BlockReportProcessingThread configurable > --- > > Key: HDFS-14553 > URL: https://issues.apache.org/jira/browse/HDFS-14553 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: HDFS-14553.001.patch, HDFS-14553.branch-3.2.patch > > > ArrayBlockingQueue size of BlockReportProcessingThread is static 1024 > currently, I propose to make this queue size configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15600) TestRouterQuota fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203365#comment-17203365
]
Íñigo Goiri commented on HDFS-15600:
Can we go with the fix the [~ayushtkn] mentioned?
This is showing up in all RBF JIRAs.
> TestRouterQuota fails in trunk
> --
>
> Key: HDFS-15600
> URL: https://issues.apache.org/jira/browse/HDFS-15600
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Ayush Saxena
>Priority: Major
>
> The test is failing due to addition of a new storage type {{NVDIMM}} in
> middle.
> Ref :
> https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/204/testReport/org.apache.hadoop.hdfs.server.federation.router/TestRouterQuota/testStorageTypeQuota/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15591) RBF: Fix webHdfs file display error
[ https://issues.apache.org/jira/browse/HDFS-15591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203360#comment-17203360 ] Íñigo Goiri commented on HDFS-15591: +1 on [^HDFS-15591-004.patch]. > RBF: Fix webHdfs file display error > --- > > Key: HDFS-15591 > URL: https://issues.apache.org/jira/browse/HDFS-15591 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > Attachments: HDFS-15591-001.patch, HDFS-15591-002.patch, > HDFS-15591-003.patch, HDFS-15591-004.patch, RBF_Browse_Directory.png, > RBF_Browse_Directory_PostFix.png, after-1.jpg, after-2.jpg, before-1.jpg, > before-2.jpg > > > The path mounted by the router does not exist on NN，router will create > virtual folder with the mount name, but the "browse the file syaytem" display > on http is wrong. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15604) Fix Typo for HdfsDataNodeAdminGuide doc
[
https://issues.apache.org/jira/browse/HDFS-15604?focusedWorklogId=492066=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-492066
]
ASF GitHub Bot logged work on HDFS-15604:
-
Author: ASF GitHub Bot
Created on: 28/Sep/20 16:54
Start Date: 28/Sep/20 16:54
Worklog Time Spent: 10m
Work Description: goiri merged pull request #2345:
URL: https://github.com/apache/hadoop/pull/2345
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 492066)
Time Spent: 20m (was: 10m)
> Fix Typo for HdfsDataNodeAdminGuide doc
> ---
>
> Key: HDFS-15604
> URL: https://issues.apache.org/jira/browse/HDFS-15604
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.3.0
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When reading the document, find the typo
> {quote}
> The node in in maintenance state
> {quote}
> It should be "The node is in maintenance state"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15604) Fix Typo for HdfsDataNodeAdminGuide doc
[
https://issues.apache.org/jira/browse/HDFS-15604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-15604:
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix Typo for HdfsDataNodeAdminGuide doc
> ---
>
> Key: HDFS-15604
> URL: https://issues.apache.org/jira/browse/HDFS-15604
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.3.0
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When reading the document, find the typo
> {quote}
> The node in in maintenance state
> {quote}
> It should be "The node is in maintenance state"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15603) RBF: Fix getLocationsForPath twice in create operation
[ https://issues.apache.org/jira/browse/HDFS-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203356#comment-17203356 ] Íñigo Goiri commented on HDFS-15603: This was introduced by HDFS-14316. I think this is correct and it's more a matter of efficiency and the existing tests should already cover it. The TestRouterQuota error is already tracked in another JIRA. +1 on [^HDFS-15603-001.patch]. > RBF: Fix getLocationsForPath twice in create operation > -- > > Key: HDFS-15603 > URL: https://issues.apache.org/jira/browse/HDFS-15603 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > Attachments: HDFS-15603-001.patch > > > getLocationsForPath in create(), but getLocationsForPath again in > getCreateLocation(),is not necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15543) RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled.
[
https://issues.apache.org/jira/browse/HDFS-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203354#comment-17203354
]
Íñigo Goiri commented on HDFS-15543:
* Can we keep {{isUnavailableSubclusterException()}} at least package protected
instead of fully visible?
* Can we make the comment in line 605 consistent with the other logs (space
before the start, capitalization, etc.)?
* Let's make {{getNameSpaceInfo()}} not modified the input but return a new one
without the namespace. Not a fan of methods that modify inputs.
> RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount
> points with fault Tolerance enabled.
>
>
> Key: HDFS-15543
> URL: https://issues.apache.org/jira/browse/HDFS-15543
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Affects Versions: 3.1.1
>Reporter: Harshakiran Reddy
>Assignee: Hemanth Boyina
>Priority: Major
> Attachments: HDFS-15543.001.patch, HDFS-15543.002.patch,
> HDFS-15543.003.patch, HDFS-15543.004.patch, HDFS-15543.005.patch,
> HDFS-15543_testrepro.patch
>
>
> A RANDOM mount point should allow to creating new files if one subcluster is
> down also with Fault Tolerance was enabled. but here it's failed.
> MultiDestination_client]# hdfs dfsrouteradmin -ls /test_ec
> *Mount Table Entries:*
> Source Destinations Owner Group Mode Quota/Usage
> /test_ec *hacluster->/tes_ec,hacluster1->/tes_ec* test ficommon rwxr-xr-x
> [NsQuota: -/-, SsQuota: -/-]
> *File Write throne the Exception:-*
> 2020-08-26 19:13:21,839 WARN hdfs.DataStreamer: Abandoning blk_1073743375_2551
> 2020-08-26 19:13:21,877 WARN hdfs.DataStreamer: Excluding datanode
> DatanodeInfoWithStorage[DISK]
> 2020-08-26 19:13:21,878 WARN hdfs.DataStreamer: DataStreamer Exception
> java.io.IOException: Unable to create new block.
> at
> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1758)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718)
> 2020-08-26 19:13:21,879 WARN hdfs.DataStreamer: Could not get block
> locations. Source file "/test_ec/f1._COPYING_" - Aborting...block==null
> put: Could not get block locations. Source file "/test_ec/f1._COPYING_" -
> Aborting...block==null
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15530) RBF: Fix typo in DFS_ROUTER_QUOTA_CACHE_UPDATE_INTERVAL var definition
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-15530: --- Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > RBF: Fix typo in DFS_ROUTER_QUOTA_CACHE_UPDATE_INTERVAL var definition > -- > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Assignee: Sha Fanghao >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15530.patch > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15530) RBF: Fix typo in DFS_ROUTER_QUOTA_CACHE_UPDATE_INTERVAL var definition
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203351#comment-17203351 ] Íñigo Goiri commented on HDFS-15530: Thanks [~lausaa] for the fix and [~hemanthboyina] for the review. > RBF: Fix typo in DFS_ROUTER_QUOTA_CACHE_UPDATE_INTERVAL var definition > -- > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Assignee: Sha Fanghao >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15530.patch > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15530) RBF: Fix typo in DFS_ROUTER_QUOTA_CACHE_UPDATE_INTERVAL var definition
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-15530: --- Summary: RBF: Fix typo in DFS_ROUTER_QUOTA_CACHE_UPDATE_INTERVAL var definition (was: RBF: Wrong word in the HDFS Router configuration file) > RBF: Fix typo in DFS_ROUTER_QUOTA_CACHE_UPDATE_INTERVAL var definition > -- > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Assignee: Sha Fanghao >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15530.patch > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15530) RBF: Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri reassigned HDFS-15530: -- Assignee: Sha Fanghao > RBF: Wrong word in the HDFS Router configuration file > - > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Assignee: Sha Fanghao >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15530.patch > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15603) RBF: Fix getLocationsForPath twice in create operation
[
https://issues.apache.org/jira/browse/HDFS-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203258#comment-17203258
]
Hadoop QA commented on HDFS-15603:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
32s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
9s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 55s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
17s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 54s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} the patch passed with JDK
[jira] [Commented] (HDFS-15025) Applying NVDIMM storage media to HDFS
[
https://issues.apache.org/jira/browse/HDFS-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203240#comment-17203240
]
YaYun Wang commented on HDFS-15025:
---
[~liuml07]，[~ayushtkn]， we have tried to make sure that {{NVDIMM}} and
{{isRAM}} will not affect the function and existing data on hadoop by
flollowing ways:
# Running test case: run test cases on
{{FsEditLogOp#SetQuotaByStorageTypeOp}}, such as {{TestWebHDFS}}, and that can
pass without failures or errors.
# Combing the code about {{FsEditLogOp＃SetQuotaByStorageTypeOp}}, including
{{DistributedFileSystem}}, {{DFSClient}}, {{NamenodeRPCServer}},
{{FSNamesystem}}, {{FSDirAttrOp}}, {{FSEditLog}} and {{FSEditLogOp,}} etc.. We
can draw a conclusion that {{ordinal()}} of StorageType which is enum is
flexible, that is, {{ordinal()}} can vary with the value of {{StorageType}} and
that's different with {{TestRouterQuota.testStorageTypeQuota}} where
{{ordinal()}} is used as the index of array.
# Verifing the function of hadoop with {{NVDIMM}} and {{isRAM}} : the
functions of DFSIO, wordcount, put/get built-in hadoop can used normally. The
old storage types and storage policies are normal too.
# Verifing the function after upgrade by existing data: in the first, we write
data to hadoop without the patch using DFSIO, wordcount and put/get, then
updata the version of hadoop with the patch, and the existing data of old
version can be accessed and used normally after upgrade.
> Applying NVDIMM storage media to HDFS
> -
>
> Key: HDFS-15025
> URL: https://issues.apache.org/jira/browse/HDFS-15025
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Reporter: YaYun Wang
>Assignee: YaYun Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch,
> HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch,
> HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch
>
> Time Spent: 12h 10m
> Remaining Estimate: 0h
>
> The non-volatile memory NVDIMM is faster than SSD, it can be used
> simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on
> NVDIMM can not only improves the response rate of HDFS, but also ensure the
> reliability of the data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15603) RBF: Fix getLocationsForPath twice in create operation
[ https://issues.apache.org/jira/browse/HDFS-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203232#comment-17203232 ] Wanqiang Ji commented on HDFS-15603: Hi [~hexiaoqiao], I found we had many UTs using this method, maybe we don't need to add new UT for this. If we should to test the `RouterClientProtocol` class, I recommend to fill new JIRA for this. Any thoughts? > RBF: Fix getLocationsForPath twice in create operation > -- > > Key: HDFS-15603 > URL: https://issues.apache.org/jira/browse/HDFS-15603 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > Attachments: HDFS-15603-001.patch > > > getLocationsForPath in create(), but getLocationsForPath again in > getCreateLocation(),is not necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15530) RBF: Wrong word in the HDFS Router configuration file
[ https://issues.apache.org/jira/browse/HDFS-15530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203229#comment-17203229 ] Sha Fanghao commented on HDFS-15530: [~elgoiri] Can this issue be assigned to me? > RBF: Wrong word in the HDFS Router configuration file > - > > Key: HDFS-15530 > URL: https://issues.apache.org/jira/browse/HDFS-15530 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sha Fanghao >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15530.patch > > > UPDATE was misspelled as UPATE in DFS_ROUTER_QUOTA_CACHE_UPATE_INTERVAL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15591) RBF: Fix webHdfs file display error
[ https://issues.apache.org/jira/browse/HDFS-15591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15591: Attachment: RBF_Browse_Directory_PostFix.png > RBF: Fix webHdfs file display error > --- > > Key: HDFS-15591 > URL: https://issues.apache.org/jira/browse/HDFS-15591 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > Attachments: HDFS-15591-001.patch, HDFS-15591-002.patch, > HDFS-15591-003.patch, HDFS-15591-004.patch, RBF_Browse_Directory.png, > RBF_Browse_Directory_PostFix.png, after-1.jpg, after-2.jpg, before-1.jpg, > before-2.jpg > > > The path mounted by the router does not exist on NN，router will create > virtual folder with the mount name, but the "browse the file syaytem" display > on http is wrong. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15570) Hadoop Erasure Coding ISA-L Check Request..
[ https://issues.apache.org/jira/browse/HDFS-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] isurika resolved HDFS-15570. Resolution: Fixed > Hadoop Erasure Coding ISA-L Check Request.. > --- > > Key: HDFS-15570 > URL: https://issues.apache.org/jira/browse/HDFS-15570 > Project: Hadoop HDFS > Issue Type: Test > Components: erasure-coding, hadoop-client >Affects Versions: 3.2.1 > Environment: *-OS Version* : CentOS Linux release 8.0.1905 (Core) > *-Hadoop Version* : hadoop-3.2.1 > >Reporter: isurika >Priority: Trivial > Fix For: 3.2.1 > > > I am testing the performance of erasure coding in Hadoop 3.2.1 version > environment. > Apply ISA-L by referring to the manual > ([https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]) > *▶ JOB LIST (Proceed in manual order)* > # ISA-L library installed on all servers > ld ISA-L library : [https://github.com/01org/isa-l] > # hadoop build source (hadoop-3.2.1-src.tar.gz / mvn package -Pdist,native > -Dtar -Drequire.isal -Drequire.snappy) > # Deploy the built hadoop-3.2.1 folder to all servers > But In Test (File upload Time/Select Orc table in Hive) > There is no improvement in performance when tested. > Ask for advice on work > *[Question 1]* > Is there anything wrong with the installation? Are there any missing tasks? > *[Question 2]* > Why doesn't it speed up ? (File upload Time/Select Orc table in Hive) > *[Question 3]* > Whether to use ISA-L in Hadoop, How to check? > ※ When I used the "hdfs ec -listCodecs" command, I expected to see ISA-L. > But no > ※ The warn log that occurs before applying ISA-L is It no longer occurs. > -- > WARN org.apache.hadoop.io.erasurecode.ErasureCodeNative: ISA-L support is > not available in your platform... using builtin-java codec where applicable > -- > > *▶Reference information* > === > *-CPU INFO (namenode :2 / datanode : 5)* > - > namenode cpu : Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz > datanode cpu : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz > - > *-hadoop checknative (After source build)* > - > Native library checking: > hadoop: true /nds/hadoop-3.2.1/lib/native/libhadoop.so.1.0.0 > zlib: true /lib64/libz.so.1 > zstd : false > snappy: true /lib64/libsnappy.so.1 > lz4: true revision:10301 > bzip2: false > openssl: true /lib64/libcrypto.so > *ISA-L: true /lib64/libisal.so.2* > - > *-hdfs ec -listCodecs* > Erasure Coding Codecs: Codec [Coder List] > RS [RS_NATIVE, RS_JAVA] > RS-LEGACY [RS-LEGACY_JAVA] > XOR [XOR_NATIVE, XOR_JAVA] > === > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15570) Hadoop Erasure Coding ISA-L Check Request..
[ https://issues.apache.org/jira/browse/HDFS-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] isurika updated HDFS-15570: --- Issue Type: Test (was: Bug) Priority: Trivial (was: Major) > Hadoop Erasure Coding ISA-L Check Request.. > --- > > Key: HDFS-15570 > URL: https://issues.apache.org/jira/browse/HDFS-15570 > Project: Hadoop HDFS > Issue Type: Test > Components: erasure-coding, hadoop-client >Affects Versions: 3.2.1 > Environment: *-OS Version* : CentOS Linux release 8.0.1905 (Core) > *-Hadoop Version* : hadoop-3.2.1 > >Reporter: isurika >Priority: Trivial > Fix For: 3.2.1 > > > I am testing the performance of erasure coding in Hadoop 3.2.1 version > environment. > Apply ISA-L by referring to the manual > ([https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]) > *▶ JOB LIST (Proceed in manual order)* > # ISA-L library installed on all servers > ld ISA-L library : [https://github.com/01org/isa-l] > # hadoop build source (hadoop-3.2.1-src.tar.gz / mvn package -Pdist,native > -Dtar -Drequire.isal -Drequire.snappy) > # Deploy the built hadoop-3.2.1 folder to all servers > But In Test (File upload Time/Select Orc table in Hive) > There is no improvement in performance when tested. > Ask for advice on work > *[Question 1]* > Is there anything wrong with the installation? Are there any missing tasks? > *[Question 2]* > Why doesn't it speed up ? (File upload Time/Select Orc table in Hive) > *[Question 3]* > Whether to use ISA-L in Hadoop, How to check? > ※ When I used the "hdfs ec -listCodecs" command, I expected to see ISA-L. > But no > ※ The warn log that occurs before applying ISA-L is It no longer occurs. > -- > WARN org.apache.hadoop.io.erasurecode.ErasureCodeNative: ISA-L support is > not available in your platform... using builtin-java codec where applicable > -- > > *▶Reference information* > === > *-CPU INFO (namenode :2 / datanode : 5)* > - > namenode cpu : Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz > datanode cpu : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz > - > *-hadoop checknative (After source build)* > - > Native library checking: > hadoop: true /nds/hadoop-3.2.1/lib/native/libhadoop.so.1.0.0 > zlib: true /lib64/libz.so.1 > zstd : false > snappy: true /lib64/libsnappy.so.1 > lz4: true revision:10301 > bzip2: false > openssl: true /lib64/libcrypto.so > *ISA-L: true /lib64/libisal.so.2* > - > *-hdfs ec -listCodecs* > Erasure Coding Codecs: Codec [Coder List] > RS [RS_NATIVE, RS_JAVA] > RS-LEGACY [RS-LEGACY_JAVA] > XOR [XOR_NATIVE, XOR_JAVA] > === > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15591) RBF: Fix webHdfs file display error
[ https://issues.apache.org/jira/browse/HDFS-15591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203210#comment-17203210 ] Ayush Saxena commented on HDFS-15591: - Thanx [~wangzhaohui] for the fix. Changes LGTM. [~elgoiri] were you able to repro this issue? > RBF: Fix webHdfs file display error > --- > > Key: HDFS-15591 > URL: https://issues.apache.org/jira/browse/HDFS-15591 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > Attachments: HDFS-15591-001.patch, HDFS-15591-002.patch, > HDFS-15591-003.patch, HDFS-15591-004.patch, RBF_Browse_Directory.png, > after-1.jpg, after-2.jpg, before-1.jpg, before-2.jpg > > > The path mounted by the router does not exist on NN，router will create > virtual folder with the mount name, but the "browse the file syaytem" display > on http is wrong. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15603) RBF: Fix getLocationsForPath twice in create operation
[ https://issues.apache.org/jira/browse/HDFS-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203199#comment-17203199 ] Xiaoqiao He commented on HDFS-15603: [~wangzhaohui] Great catch here. Would you like to add unit test to verify this changes? > RBF: Fix getLocationsForPath twice in create operation > -- > > Key: HDFS-15603 > URL: https://issues.apache.org/jira/browse/HDFS-15603 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > Attachments: HDFS-15603-001.patch > > > getLocationsForPath in create(), but getLocationsForPath again in > getCreateLocation(),is not necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15604) Fix Typo for HdfsDataNodeAdminGuide doc
[
https://issues.apache.org/jira/browse/HDFS-15604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-15604:
--
Labels: pull-request-available (was: )
> Fix Typo for HdfsDataNodeAdminGuide doc
> ---
>
> Key: HDFS-15604
> URL: https://issues.apache.org/jira/browse/HDFS-15604
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.3.0
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Trivial
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When reading the document, find the typo
> {quote}
> The node in in maintenance state
> {quote}
> It should be "The node is in maintenance state"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15604) Fix Typo for HdfsDataNodeAdminGuide doc
[
https://issues.apache.org/jira/browse/HDFS-15604?focusedWorklogId=491957=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-491957
]
ASF GitHub Bot logged work on HDFS-15604:
-
Author: ASF GitHub Bot
Created on: 28/Sep/20 12:41
Start Date: 28/Sep/20 12:41
Worklog Time Spent: 10m
Work Description: ferhui opened a new pull request #2345:
URL: https://github.com/apache/hadoop/pull/2345
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.)
For more details, please see
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 491957)
Remaining Estimate: 0h
Time Spent: 10m
> Fix Typo for HdfsDataNodeAdminGuide doc
> ---
>
> Key: HDFS-15604
> URL: https://issues.apache.org/jira/browse/HDFS-15604
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.3.0
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Trivial
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When reading the document, find the typo
> {quote}
> The node in in maintenance state
> {quote}
> It should be "The node is in maintenance state"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15604) Fix Typo for HdfsDataNodeAdminGuide doc
[
https://issues.apache.org/jira/browse/HDFS-15604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Fei updated HDFS-15604:
---
Status: Patch Available (was: Open)
> Fix Typo for HdfsDataNodeAdminGuide doc
> ---
>
> Key: HDFS-15604
> URL: https://issues.apache.org/jira/browse/HDFS-15604
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.3.0
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Trivial
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When reading the document, find the typo
> {quote}
> The node in in maintenance state
> {quote}
> It should be "The node is in maintenance state"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15604) Fix Typo for HdfsDataNodeAdminGuide doc
[
https://issues.apache.org/jira/browse/HDFS-15604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Fei updated HDFS-15604:
---
Component/s: documentation
> Fix Typo for HdfsDataNodeAdminGuide doc
> ---
>
> Key: HDFS-15604
> URL: https://issues.apache.org/jira/browse/HDFS-15604
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.3.0
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Trivial
>
> When reading the document, find the typo
> {quote}
> The node in in maintenance state
> {quote}
> It should be "The node is in maintenance state"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15604) Fix Typo for HdfsDataNodeAdminGuide doc
[
https://issues.apache.org/jira/browse/HDFS-15604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Fei updated HDFS-15604:
---
Description:
When reading the document, find the typo
{quote}
The node in in maintenance state
{quote}
It should be "The node is in maintenance state"
> Fix Typo for HdfsDataNodeAdminGuide doc
> ---
>
> Key: HDFS-15604
> URL: https://issues.apache.org/jira/browse/HDFS-15604
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Trivial
>
> When reading the document, find the typo
> {quote}
> The node in in maintenance state
> {quote}
> It should be "The node is in maintenance state"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15604) Fix Typo for HdfsDataNodeAdminGuide doc
Hui Fei created HDFS-15604:
--
Summary: Fix Typo for HdfsDataNodeAdminGuide doc
Key: HDFS-15604
URL: https://issues.apache.org/jira/browse/HDFS-15604
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 3.3.0
Environment: When reading the document, find the typo
{quote}
The node in in maintenance state
{quote}
It should be "The node is in maintenance state"
Reporter: Hui Fei
Assignee: Hui Fei
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15604) Fix Typo for HdfsDataNodeAdminGuide doc
[
https://issues.apache.org/jira/browse/HDFS-15604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Fei updated HDFS-15604:
---
Environment: (was: When reading the document, find the typo
{quote}
The node in in maintenance state
{quote}
It should be "The node is in maintenance state")
> Fix Typo for HdfsDataNodeAdminGuide doc
> ---
>
> Key: HDFS-15604
> URL: https://issues.apache.org/jira/browse/HDFS-15604
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Trivial
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15603) RBF: Fix getLocationsForPath twice in create operation
[ https://issues.apache.org/jira/browse/HDFS-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangzhaohui updated HDFS-15603: --- Status: Patch Available (was: Open) > RBF: Fix getLocationsForPath twice in create operation > -- > > Key: HDFS-15603 > URL: https://issues.apache.org/jira/browse/HDFS-15603 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > Attachments: HDFS-15603-001.patch > > > getLocationsForPath in create(), but getLocationsForPath again in > getCreateLocation(),is not necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15603) RBF: Fix getLocationsForPath twice in create operation
[ https://issues.apache.org/jira/browse/HDFS-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangzhaohui updated HDFS-15603: --- Description: getLocationsForPath in create(), but getLocationsForPath again in getCreateLocation(),is not necessary. (was: getLocationsForPath in create(), but getLocationsForPath again in getCreateLocation(),is not necessary) > RBF: Fix getLocationsForPath twice in create operation > -- > > Key: HDFS-15603 > URL: https://issues.apache.org/jira/browse/HDFS-15603 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > Attachments: HDFS-15603-001.patch > > > getLocationsForPath in create(), but getLocationsForPath again in > getCreateLocation(),is not necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15603) RBF: Fix getLocationsForPath twice in create operation
[ https://issues.apache.org/jira/browse/HDFS-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangzhaohui updated HDFS-15603: --- Description: getLocationsForPath in create(), but getLocationsForPath again in getCreateLocation(),is not necessary > RBF: Fix getLocationsForPath twice in create operation > -- > > Key: HDFS-15603 > URL: https://issues.apache.org/jira/browse/HDFS-15603 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > Attachments: HDFS-15603-001.patch > > > getLocationsForPath in create(), but getLocationsForPath again in > getCreateLocation(),is not necessary -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15603) RBF: Fix getLocationsForPath twice in create operation
[ https://issues.apache.org/jira/browse/HDFS-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangzhaohui updated HDFS-15603: --- Attachment: HDFS-15603-001.patch > RBF: Fix getLocationsForPath twice in create operation > -- > > Key: HDFS-15603 > URL: https://issues.apache.org/jira/browse/HDFS-15603 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > Attachments: HDFS-15603-001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15603) RBF: Fix getLocationsForPath twice in create operation
wangzhaohui created HDFS-15603: -- Summary: RBF: Fix getLocationsForPath twice in create operation Key: HDFS-15603 URL: https://issues.apache.org/jira/browse/HDFS-15603 Project: Hadoop HDFS Issue Type: Improvement Reporter: wangzhaohui -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15603) RBF: Fix getLocationsForPath twice in create operation
[ https://issues.apache.org/jira/browse/HDFS-15603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangzhaohui reassigned HDFS-15603: -- Assignee: wangzhaohui > RBF: Fix getLocationsForPath twice in create operation > -- > > Key: HDFS-15603 > URL: https://issues.apache.org/jira/browse/HDFS-15603 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: wangzhaohui >Assignee: wangzhaohui >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15548) Allow configuring DISK/ARCHIVE storage types on same device mount
[ https://issues.apache.org/jira/browse/HDFS-15548?focusedWorklogId=491902=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-491902 ] ASF GitHub Bot logged work on HDFS-15548: - Author: ASF GitHub Bot Created on: 28/Sep/20 09:46 Start Date: 28/Sep/20 09:46 Worklog Time Spent: 10m Work Description: Hexiaoqiao commented on pull request #2288: URL: https://github.com/apache/hadoop/pull/2288#issuecomment-699901780 +1 from my side. Ping @jojochuang @ayushtkn @goiri would you like to have another check? Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 491902) Time Spent: 5h 50m (was: 5h 40m) > Allow configuring DISK/ARCHIVE storage types on same device mount > - > > Key: HDFS-15548 > URL: https://issues.apache.org/jira/browse/HDFS-15548 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Major > Labels: pull-request-available > Time Spent: 5h 50m > Remaining Estimate: 0h > > We can allow configuring DISK/ARCHIVE storage types on the same device mount > on two separate directories. > Users should be able to configure the capacity for each. Also, the datanode > usage report should report stats correctly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15548) Allow configuring DISK/ARCHIVE storage types on same device mount
[
https://issues.apache.org/jira/browse/HDFS-15548?focusedWorklogId=491898=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-491898
]
ASF GitHub Bot logged work on HDFS-15548:
-
Author: ASF GitHub Bot
Created on: 28/Sep/20 09:41
Start Date: 28/Sep/20 09:41
Worklog Time Spent: 10m
Work Description: Hexiaoqiao commented on a change in pull request #2288:
URL: https://github.com/apache/hadoop/pull/2288#discussion_r495813990
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
##
@@ -412,16 +435,28 @@ long getBlockPoolUsed(String bpid) throws IOException {
*/
@VisibleForTesting
public long getCapacity() {
+long capacity;
if (configuredCapacity < 0L) {
long remaining;
if (cachedCapacity > 0L) {
remaining = cachedCapacity - getReserved();
} else {
remaining = usage.getCapacity() - getReserved();
}
- return Math.max(remaining, 0L);
+ capacity = Math.max(remaining, 0L);
+} else {
+ capacity = configuredCapacity;
+}
+
+if (enableSameDiskArchival) {
Review comment:
Thanks @LeonGao91 for your detailed comments. It makes sense for me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 491898)
Time Spent: 5h 40m (was: 5.5h)
> Allow configuring DISK/ARCHIVE storage types on same device mount
> -
>
> Key: HDFS-15548
> URL: https://issues.apache.org/jira/browse/HDFS-15548
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Leon Gao
>Assignee: Leon Gao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> We can allow configuring DISK/ARCHIVE storage types on the same device mount
> on two separate directories.
> Users should be able to configure the capacity for each. Also, the datanode
> usage report should report stats correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15577) Refactor TestTracing
[ https://issues.apache.org/jira/browse/HDFS-15577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-15577: Fix Version/s: 3.4.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Refactor TestTracing > > > Key: HDFS-15577 > URL: https://issues.apache.org/jira/browse/HDFS-15577 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > There are some unused imports, unused variables, and checkstyle warnings. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15577) Refactor TestTracing
[ https://issues.apache.org/jira/browse/HDFS-15577?focusedWorklogId=491886=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-491886 ] ASF GitHub Bot logged work on HDFS-15577: - Author: ASF GitHub Bot Created on: 28/Sep/20 09:27 Start Date: 28/Sep/20 09:27 Worklog Time Spent: 10m Work Description: tasanuma commented on pull request #2302: URL: https://github.com/apache/hadoop/pull/2302#issuecomment-699892268 @aajisaka Thanks for your review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 491886) Time Spent: 0.5h (was: 20m) > Refactor TestTracing > > > Key: HDFS-15577 > URL: https://issues.apache.org/jira/browse/HDFS-15577 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > There are some unused imports, unused variables, and checkstyle warnings. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15577) Refactor TestTracing
[ https://issues.apache.org/jira/browse/HDFS-15577?focusedWorklogId=491887=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-491887 ] ASF GitHub Bot logged work on HDFS-15577: - Author: ASF GitHub Bot Created on: 28/Sep/20 09:27 Start Date: 28/Sep/20 09:27 Worklog Time Spent: 10m Work Description: tasanuma merged pull request #2302: URL: https://github.com/apache/hadoop/pull/2302 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 491887) Time Spent: 40m (was: 0.5h) > Refactor TestTracing > > > Key: HDFS-15577 > URL: https://issues.apache.org/jira/browse/HDFS-15577 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > There are some unused imports, unused variables, and checkstyle warnings. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15098: -- Labels: pull-request-available sm4 (was: sm4) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: liusheng >Priority: Major > Labels: pull-request-available, sm4 > Fix For: 3.4.0 > > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch, > HDFS-15098.009.patch, image-2020-08-19-16-54-41-341.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?focusedWorklogId=491875=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-491875 ] ASF GitHub Bot logged work on HDFS-15098: - Author: ASF GitHub Bot Created on: 28/Sep/20 08:53 Start Date: 28/Sep/20 08:53 Worklog Time Spent: 10m Work Description: liusheng closed pull request #2211: URL: https://github.com/apache/hadoop/pull/2211 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 491875) Time Spent: 40m (was: 0.5h) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: liusheng >Priority: Major > Labels: pull-request-available, sm4 > Fix For: 3.4.0 > > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch, > HDFS-15098.009.patch, image-2020-08-19-16-54-41-341.png > > Time Spent: 40m > Remaining Estimate: 0h > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?focusedWorklogId=491874=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-491874 ] ASF GitHub Bot logged work on HDFS-15098: - Author: ASF GitHub Bot Created on: 28/Sep/20 08:52 Start Date: 28/Sep/20 08:52 Worklog Time Spent: 10m Work Description: liusheng commented on pull request #2211: URL: https://github.com/apache/hadoop/pull/2211#issuecomment-699873817 the patch in JIRA issue has been merged, abandon this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 491874) Time Spent: 0.5h (was: 20m) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: liusheng >Priority: Major > Labels: sm4 > Fix For: 3.4.0 > > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch, > HDFS-15098.009.patch, image-2020-08-19-16-54-41-341.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12861) Track speed in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-12861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203063#comment-17203063
]
Hadoop QA commented on HDFS-12861:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s{color}
| {color:red}{color} | {color:red} HDFS-12861 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-12861 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918330/HDFS-12861-10-april-18.patch
|
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/209/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Track speed in DFSClient
>
>
> Key: HDFS-12861
> URL: https://issues.apache.org/jira/browse/HDFS-12861
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: María Fernanda Borge
>Priority: Major
> Attachments: HDFS-12861-10-april-18.patch
>
>
> Sometimes we get slow jobs because of the access to HDFS. However, is hard to
> tell what is the actual speed. We propose to add a log line with something
> like:
> {code}
> 2017-11-19 09:55:26,309 INFO [main] hdfs.DFSClient: blk_1107222019_38144502
> READ 129500B in 7ms 17.6MB/s
> 2017-11-27 19:01:04,141 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792057_86833357 WRITE 131072B in 10ms 12.5MB/s
> 2017-11-27 19:01:14,219 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792069_86833369 WRITE 131072B in 12ms 10.4MB/s
> 2017-11-27 19:01:24,282 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792081_86833381 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:34,330 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792093_86833393 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:44,408 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792105_86833405 WRITE 131072B in 11ms 11.4MB/s
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12861) Track speed in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-12861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203054#comment-17203054
]
huhaiyang commented on HDFS-12861:
--
[~elgoiri]It looks like very good work And are there plans to merge into the
trunk? Thanks.
> Track speed in DFSClient
>
>
> Key: HDFS-12861
> URL: https://issues.apache.org/jira/browse/HDFS-12861
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: María Fernanda Borge
>Priority: Major
> Attachments: HDFS-12861-10-april-18.patch
>
>
> Sometimes we get slow jobs because of the access to HDFS. However, is hard to
> tell what is the actual speed. We propose to add a log line with something
> like:
> {code}
> 2017-11-19 09:55:26,309 INFO [main] hdfs.DFSClient: blk_1107222019_38144502
> READ 129500B in 7ms 17.6MB/s
> 2017-11-27 19:01:04,141 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792057_86833357 WRITE 131072B in 10ms 12.5MB/s
> 2017-11-27 19:01:14,219 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792069_86833369 WRITE 131072B in 12ms 10.4MB/s
> 2017-11-27 19:01:24,282 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792081_86833381 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:34,330 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792093_86833393 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:44,408 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792105_86833405 WRITE 131072B in 11ms 11.4MB/s
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12861) Track speed in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-12861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203054#comment-17203054
]
huhaiyang edited comment on HDFS-12861 at 9/28/20, 7:29 AM:
[~elgoiri] It looks like very good work And are there plans to merge into the
trunk? Thanks.
was (Author: haiyang hu):
[~elgoiri]It looks like very good work And are there plans to merge into the
trunk? Thanks.
> Track speed in DFSClient
>
>
> Key: HDFS-12861
> URL: https://issues.apache.org/jira/browse/HDFS-12861
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: María Fernanda Borge
>Priority: Major
> Attachments: HDFS-12861-10-april-18.patch
>
>
> Sometimes we get slow jobs because of the access to HDFS. However, is hard to
> tell what is the actual speed. We propose to add a log line with something
> like:
> {code}
> 2017-11-19 09:55:26,309 INFO [main] hdfs.DFSClient: blk_1107222019_38144502
> READ 129500B in 7ms 17.6MB/s
> 2017-11-27 19:01:04,141 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792057_86833357 WRITE 131072B in 10ms 12.5MB/s
> 2017-11-27 19:01:14,219 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792069_86833369 WRITE 131072B in 12ms 10.4MB/s
> 2017-11-27 19:01:24,282 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792081_86833381 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:34,330 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792093_86833393 WRITE 131072B in 11ms 11.4MB/s
> 2017-11-27 19:01:44,408 INFO [DataStreamer for file
> /hdfs-federation/stats/2017/11/27/151183800.json] hdfs.DFSClient:
> blk_1135792105_86833405 WRITE 131072B in 11ms 11.4MB/s
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14553) Make queue size of BlockReportProcessingThread configurable
[
https://issues.apache.org/jira/browse/HDFS-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203018#comment-17203018
]
Xiaoqiao He commented on HDFS-14553:
[~xyao] Thanks for your comments.
{quote}does the new configurable block report queue help mitigate the "Block
report queue is full" issue?{quote}
I think 'Block report queue is full' mentioned here is about NameNode restart
and Block Report flood， right? If that I think it could be helpful when change
this config. I want to state that HDFS-7923 could be another better solution.
In my practice, I set this queue size to 4096. the queue's load is still very
high when NameNode restart, But not seem any issues when process IBR. Thanks.
> Make queue size of BlockReportProcessingThread configurable
> ---
>
> Key: HDFS-14553
> URL: https://issues.apache.org/jira/browse/HDFS-14553
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Xiaoqiao He
>Assignee: Xiaoqiao He
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14553.001.patch, HDFS-14553.branch-3.2.patch
>
>
> ArrayBlockingQueue size of BlockReportProcessingThread is static 1024
> currently, I propose to make this queue size configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
