[jira] [Commented] (HDFS-13035) Owner should be allowed to set xattr if not already set.
[
https://issues.apache.org/jira/browse/HDFS-13035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337089#comment-16337089
]
Xiao Chen commented on HDFS-13035:
--
[~andrew.wang] obviously has more weight to comment.
The proposed relaxation of a+b for file owner to set feinfo SGTM.
Question: what if the owner typo'ed and set a wrong xattr, then wanted to
remove? Do we relax it for {{removeXAttr}} and \{{getXAttrs}} as well (which
sounds fine too for xattr in general, but less so for fe info...)?
> Owner should be allowed to set xattr if not already set.
>
>
> Key: HDFS-13035
> URL: https://issues.apache.org/jira/browse/HDFS-13035
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Reporter: Rushabh S Shah
>Priority: Major
>
> Motivation: This is needed to support encryption zones on WebhdfsFileSystem.
> For writing into EZ directory, webhdfs client will encrypt data on client
> side and will always write into /.reserved/raw/ directory so that datanode
> will not encrypt since its writing to /.reserved/raw directory.
> But then we have to somehow set crypto related x-attrs on the file.
> Currently only super user is allowed to set x-attrs.
> So I am proposing that the file owner(and superuser) should be allowed to set
> crypto related x-attrs if they are already not set.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337079#comment-16337079 ] Rakesh R commented on HDFS-10285: - Thank you all contributors in making this feature. I have rebased HDFS-10285 branch to the latest trunk code and attaching consolidated patch to the umbrella jira to get the QA report. > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-10285-consolidated-merge-patch-02.patch, > HDFS-10285-consolidated-merge-patch-03.patch, > HDFS-10285-consolidated-merge-patch-04.patch, > HDFS-SPS-TestReport-20170708.pdf, SPS Modularization.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf, > Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12974) Exception information can not be returned when I create transparent encryption zone.
[
https://issues.apache.org/jira/browse/HDFS-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337071#comment-16337071
]
Xiao Chen commented on HDFS-12974:
--
Sorry for the late response, I was on PTO for 2 weeks and have been catching up
this week.
I don't feel strongly here. Agree with [~shahrs87] that fixing this inside
{{AuthorizationException}} would be cleaner. Not sure whether there would be
compatibility concerns on the behavior (compat guidelines doesn't cover this
AFAICT). This also makes {{getStackTrace}} and {{printStackTrace}} behave a
little differently.
But given this is just a message improvement and brings no security concerns on
the exception itself, I'm +1 on fixing this in {{AuthorizationException}} as
Rushabh suggested.
> Exception information can not be returned when I create transparent
> encryption zone.
>
>
> Key: HDFS-12974
> URL: https://issues.apache.org/jira/browse/HDFS-12974
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 3.0.0
>Reporter: fang zhenyi
>Assignee: fang zhenyi
>Priority: Minor
> Attachments: HDFS-12974.001.patch, HDFS-12974.002.patch,
> HDFS-12974.003.patch, HDFS-12974.004.patch, HDFS-12974.005.patch,
> HDFS-12974.006.patch
>
>
> When I add the following configuration to the kms-acl.xml file, I create
> encrypted space and I can not get any exception information.
>
> key.acl.key2.GENERATE_EEK
> mr
>
> root@fangzhenyi01:~# hdfs crypto -createZone -keyName key2 -path /zone
> 2018-01-02 10:41:44,632 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> RemoteException:
> root@fangzhenyi01:~#
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-10285: Attachment: HDFS-10285-consolidated-merge-patch-04.patch > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-10285-consolidated-merge-patch-02.patch, > HDFS-10285-consolidated-merge-patch-03.patch, > HDFS-10285-consolidated-merge-patch-04.patch, > HDFS-SPS-TestReport-20170708.pdf, SPS Modularization.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf, > Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13051) dead lock occurs when rolleditlog rpc call happen and editPendingQ is full
[ https://issues.apache.org/jira/browse/HDFS-13051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangwei updated HDFS-13051: Target Version/s: 3.0.0, 2.7.5, 2.8.2, 2.8.1 (was: 2.8.1, 2.8.2, 2.7.5, 3.0.0) Description: When we apply AsyncEditlog in our prod env, in case, deadlock will occur and crash the namenode. in the case, rollEditlog rpc call happen while editPendingQ in AsyncEditlog is full (was: When we apply AsyncEditlog in our prod env, in extremely case, deadlock will occur and crash the namenode. in the case, rollEditlog rpc call happen while AsyncEditlog asynchronized takes edit object from queue and is going to acquire AsyncEditlog object lock to executed it's logEdit method.) Summary: dead lock occurs when rolleditlog rpc call happen and editPendingQ is full (was: dead lock occurs when using async editlog while rolleditlog rpc call happen) > dead lock occurs when rolleditlog rpc call happen and editPendingQ is full > -- > > Key: HDFS-13051 > URL: https://issues.apache.org/jira/browse/HDFS-13051 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.5 >Reporter: zhangwei >Priority: Major > > When we apply AsyncEditlog in our prod env, in case, deadlock will occur and > crash the namenode. in the case, rollEditlog rpc call happen while > editPendingQ in AsyncEditlog is full -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13048) LowRedundancyReplicatedBlocks metric can be negative
[
https://issues.apache.org/jira/browse/HDFS-13048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337047#comment-16337047
]
Takanobu Asanuma commented on HDFS-13048:
-
+1(non-binding), pending Jenkins.
> LowRedundancyReplicatedBlocks metric can be negative
>
>
> Key: HDFS-13048
> URL: https://issues.apache.org/jira/browse/HDFS-13048
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: metrics
>Affects Versions: 3.0.0
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-13048-sample.patch, HDFS-13048.001.patch
>
>
> I'm seeing {{LowRedundancyReplicatedBlocks}} become negative. This should be
> 0 or positive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13046) consider load of datanodes when read blocks of file
[ https://issues.apache.org/jira/browse/HDFS-13046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337003#comment-16337003 ] hu xiaodong edited comment on HDFS-13046 at 1/24/18 6:56 AM: - Or we can consider the load after sorting by distance. After sorting by distance, if the datanode's load is greater than the average load * considerLoadFactor, then move the datanode to the end of the list. I submitted [^HDFS-13046-considerLoadAfterSortBydistance-001-sample.patch],welcome to discuss. was (Author: xiaodong.hu): Or we can consider the load after sorting by distance. After sorting by distance, if the datanode's load is greater than the average load * considerLoadFactor, then move the datanode to the end of the list. I submitted [^HDFS-13046-considerLoadAfterSortBydistance-001.patch],welcome to discuss. > consider load of datanodes when read blocks of file > --- > > Key: HDFS-13046 > URL: https://issues.apache.org/jira/browse/HDFS-13046 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Minor > Attachments: > HDFS-13046-considerLoadAfterSortBydistance-001-sample.patch, > HDFS-13046-sample.patch > > > When sorting block locations, we just consider the distance of datanodes. can > we consider the load of datanodes? We can add a configuration such as > 'dfs.namenode.reading.considerLoad', if set to true, then sort the > blocklocations by load of the datanodes, otherwise sort by distance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13046) consider load of datanodes when read blocks of file
[ https://issues.apache.org/jira/browse/HDFS-13046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hu xiaodong updated HDFS-13046: --- Attachment: HDFS-13046-considerLoadAfterSortBydistance-001-sample.patch > consider load of datanodes when read blocks of file > --- > > Key: HDFS-13046 > URL: https://issues.apache.org/jira/browse/HDFS-13046 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Minor > Attachments: > HDFS-13046-considerLoadAfterSortBydistance-001-sample.patch, > HDFS-13046-sample.patch > > > When sorting block locations, we just consider the distance of datanodes. can > we consider the load of datanodes? We can add a configuration such as > 'dfs.namenode.reading.considerLoad', if set to true, then sort the > blocklocations by load of the datanodes, otherwise sort by distance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13046) consider load of datanodes when read blocks of file
[ https://issues.apache.org/jira/browse/HDFS-13046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hu xiaodong updated HDFS-13046: --- Attachment: (was: HDFS-13046-considerLoadAfterSortBydistance-001.patch) > consider load of datanodes when read blocks of file > --- > > Key: HDFS-13046 > URL: https://issues.apache.org/jira/browse/HDFS-13046 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Minor > Attachments: > HDFS-13046-considerLoadAfterSortBydistance-001-sample.patch, > HDFS-13046-sample.patch > > > When sorting block locations, we just consider the distance of datanodes. can > we consider the load of datanodes? We can add a configuration such as > 'dfs.namenode.reading.considerLoad', if set to true, then sort the > blocklocations by load of the datanodes, otherwise sort by distance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13046) consider load of datanodes when read blocks of file
[ https://issues.apache.org/jira/browse/HDFS-13046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337003#comment-16337003 ] hu xiaodong edited comment on HDFS-13046 at 1/24/18 6:43 AM: - Or we can consider the load after sorting by distance. After sorting by distance, if the datanode's load is greater than the average load * considerLoadFactor, then move the datanode to the end of the list. I submitted [^HDFS-13046-considerLoadAfterSortBydistance-001.patch],welcome to discuss. was (Author: xiaodong.hu): Or we can consider the load after sorting by distance. After sorting by distance, if the datanode's load is greater than the average load * considerLoadFactor, then move the datanode to the end of the list. I submitted HDFS-13046-considerLoadAfterSortBydistance-001.patch,welcome to discuss. > consider load of datanodes when read blocks of file > --- > > Key: HDFS-13046 > URL: https://issues.apache.org/jira/browse/HDFS-13046 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Minor > Attachments: HDFS-13046-considerLoadAfterSortBydistance-001.patch, > HDFS-13046-sample.patch > > > When sorting block locations, we just consider the distance of datanodes. can > we consider the load of datanodes? We can add a configuration such as > 'dfs.namenode.reading.considerLoad', if set to true, then sort the > blocklocations by load of the datanodes, otherwise sort by distance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13046) consider load of datanodes when read blocks of file
[ https://issues.apache.org/jira/browse/HDFS-13046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337003#comment-16337003 ] hu xiaodong edited comment on HDFS-13046 at 1/24/18 6:42 AM: - Or we can consider the load after sorting by distance. After sorting by distance, if the datanode's load is greater than the average load * considerLoadFactor, then move the datanode to the end of the list. I submitted HDFS-13046-considerLoadAfterSortBydistance-001.patch,welcome to discuss. was (Author: xiaodong.hu): Or we can consider the load after sorting by distance. After sorting by distance, if the datanode's load is greater than the average load * considerLoadFactor, then move the datanode to the end of the list. > consider load of datanodes when read blocks of file > --- > > Key: HDFS-13046 > URL: https://issues.apache.org/jira/browse/HDFS-13046 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Minor > Attachments: HDFS-13046-considerLoadAfterSortBydistance-001.patch, > HDFS-13046-sample.patch > > > When sorting block locations, we just consider the distance of datanodes. can > we consider the load of datanodes? We can add a configuration such as > 'dfs.namenode.reading.considerLoad', if set to true, then sort the > blocklocations by load of the datanodes, otherwise sort by distance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13048) LowRedundancyReplicatedBlocks metric can be negative
[
https://issues.apache.org/jira/browse/HDFS-13048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-13048:
-
Target Version/s: 3.1.0, 3.0.1
Status: Patch Available (was: Open)
The calculation itself is okay, however, blockStat is not actually decremented
correctly.
Attached a 001 patch to fix it.
> LowRedundancyReplicatedBlocks metric can be negative
>
>
> Key: HDFS-13048
> URL: https://issues.apache.org/jira/browse/HDFS-13048
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: metrics
>Affects Versions: 3.0.0
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-13048-sample.patch, HDFS-13048.001.patch
>
>
> I'm seeing {{LowRedundancyReplicatedBlocks}} become negative. This should be
> 0 or positive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13046) consider load of datanodes when read blocks of file
[ https://issues.apache.org/jira/browse/HDFS-13046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hu xiaodong updated HDFS-13046: --- Attachment: HDFS-13046-considerLoadAfterSortBydistance-001.patch > consider load of datanodes when read blocks of file > --- > > Key: HDFS-13046 > URL: https://issues.apache.org/jira/browse/HDFS-13046 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Minor > Attachments: HDFS-13046-considerLoadAfterSortBydistance-001.patch, > HDFS-13046-sample.patch > > > When sorting block locations, we just consider the distance of datanodes. can > we consider the load of datanodes? We can add a configuration such as > 'dfs.namenode.reading.considerLoad', if set to true, then sort the > blocklocations by load of the datanodes, otherwise sort by distance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13048) LowRedundancyReplicatedBlocks metric can be negative
[
https://issues.apache.org/jira/browse/HDFS-13048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-13048:
-
Attachment: HDFS-13048.001.patch
> LowRedundancyReplicatedBlocks metric can be negative
>
>
> Key: HDFS-13048
> URL: https://issues.apache.org/jira/browse/HDFS-13048
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: metrics
>Affects Versions: 3.0.0
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-13048-sample.patch, HDFS-13048.001.patch
>
>
> I'm seeing {{LowRedundancyReplicatedBlocks}} become negative. This should be
> 0 or positive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13046) consider load of datanodes when read blocks of file
[ https://issues.apache.org/jira/browse/HDFS-13046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337003#comment-16337003 ] hu xiaodong commented on HDFS-13046: Or we can consider the load after sorting by distance. After sorting by distance, if the datanode's load is greater than the average load * considerLoadFactor, then move the datanode to the end of the list. > consider load of datanodes when read blocks of file > --- > > Key: HDFS-13046 > URL: https://issues.apache.org/jira/browse/HDFS-13046 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Minor > Attachments: HDFS-13046-sample.patch > > > When sorting block locations, we just consider the distance of datanodes. can > we consider the load of datanodes? We can add a configuration such as > 'dfs.namenode.reading.considerLoad', if set to true, then sort the > blocklocations by load of the datanodes, otherwise sort by distance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13042) RBF: Heartbeat Router State
[
https://issues.apache.org/jira/browse/HDFS-13042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336978#comment-16336978
]
genericqa commented on HDFS-13042:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 7 new + 431 unchanged - 0 fixed = 438 total (was 431) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}117m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}168m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13042 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907417/HDFS-13042.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 8e44c7fd05b1 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 95743c6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/22784/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22784/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22784/testReport/ |
| Max. process+thread count | 3002 (vs. ulimit of 5000) |
|
[jira] [Commented] (HDFS-12522) Ozone: Remove the Priority Queues used in the Container State Manager
[
https://issues.apache.org/jira/browse/HDFS-12522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336935#comment-16336935
]
genericqa commented on HDFS-12522:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
50s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 50s{color}
| {color:red} hadoop-hdfs-project generated 12 new + 434 unchanged - 0 fixed =
446 total (was 434) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-hdfs-project: The patch generated 7 new +
9 unchanged - 0 fixed = 16 total (was 9) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
11s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
57s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-client generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
42s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}160m 58s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}226m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Long is incompatible with expected argument type ContainerID in
org.apache.hadoop.ozone.scm.container.ContainerStates.ContainerStateMap.updateContainerInfo(ContainerInfo)
At ContainerStateMap.java:argument type ContainerID in
[jira] [Commented] (HDFS-10419) Building HDFS on top of Ozone's storage containers
[
https://issues.apache.org/jira/browse/HDFS-10419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336903#comment-16336903
]
Sanjay Radia commented on HDFS-10419:
-
{quote}As a side note, I didn't understand why you used 50MB blocks in your
math. The default is 128MB and many people run HDFS with 512MB blocks.
{quote}
While default block size is 128MB in many clusters including the ones at Yahoo
(at least in 2011 when I left) the actual average block size was 50MB because
most files had one block and even the first block was not full.
> Building HDFS on top of Ozone's storage containers
> --
>
> Key: HDFS-10419
> URL: https://issues.apache.org/jira/browse/HDFS-10419
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jing Zhao
>Assignee: Jing Zhao
>Priority: Major
> Attachments: Evolving NN using new block-container layer.pdf
>
>
> In HDFS-7240, Ozone defines storage containers to store both the data and the
> metadata. The storage container layer provides an object storage interface
> and aims to manage data/metadata in a distributed manner. More details about
> storage containers can be found in the design doc in HDFS-7240.
> HDFS can adopt the storage containers to store and manage blocks. The general
> idea is:
> # Each block can be treated as an object and the block ID is the object's key.
> # Blocks will still be stored in DataNodes but as objects in storage
> containers.
> # The block management work can be separated out of the NameNode and will be
> handled by the storage container layer in a more distributed way. The
> NameNode will only manage the namespace (i.e., files and directories).
> # For each file, the NameNode only needs to record a list of block IDs which
> are used as keys to obtain real data from storage containers.
> # A new DFSClient implementation talks to both NameNode and the storage
> container layer to read/write.
> HDFS, especially the NameNode, can get much better scalability from this
> design. Currently the NameNode's heaviest workload comes from the block
> management, which includes maintaining the block-DataNode mapping, receiving
> full/incremental block reports, tracking block states (under/over/miss
> replicated), and joining every writing pipeline protocol to guarantee the
> data consistency. These work bring high memory footprint and make NameNode
> suffer from GC. HDFS-5477 already proposes to convert BlockManager as a
> service. If we can build HDFS on top of the storage container layer, we not
> only separate out the BlockManager from the NameNode, but also replace it
> with a new distributed management scheme.
> The storage container work is currently in progress in HDFS-7240, and the
> work proposed here is still in an experimental/exploring stage. We can do
> this experiment in a feature branch so that people with interests can be
> involved.
> A design doc will be uploaded later explaining more details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-12528:
-
Status: Patch Available (was: In Progress)
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HDFS-12528.000.patch, blocks_16_12_chars-20180123.png
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-12528:
-
Attachment: blocks_16_12_chars-20180123.png
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HDFS-12528.000.patch, blocks_16_12_chars-20180123.png
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336895#comment-16336895
]
Xiao Chen commented on HDFS-12528:
--
Attaching a patch to do just #1, with enhanced unit tests from John to test
behavior for the default (disable for 10 minutes) and the never disable (0).
Reason didn't do #2 is I think the threshold is really a subjective thing, and
it feels to me having #1 would be sufficient for the requests so far from this
jira.
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HDFS-12528.000.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
[
https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336852#comment-16336852
]
Dennis Huo commented on HDFS-13056:
---
Attached a quick and dirty poc patch which just hard-codes using the new
COMPOSITE-CRC approach and hard codes CRC32C to prototype this feature against
branch-2.8 (happen to have more 2.8 clusters handy for testing purposes).
Verified the concept works by generating random data, putting in HDFS with
different block sizes and chunk sizes, verified different MD5MD5CRC checksums,
and then verified that the composite CRC returns the same file-level value for
all.
Before:
{code:java}
$ hadoop fs -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmp/random-crctest.dat
$ hadoop fs -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmp/random-crctest2.dat
$ hadoop fs -Ddfs.bytes-per-checksum=1024 -cp
gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmp/random-crctest3.dat
$ hadoop fs -Ddfs.blocksize=67108864 -cp
gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmp/random-crctest4.dat
$ hadoop fs -checksum hdfs:///tmp/random-crctest*.dat
hdfs:///tmp/random-crctest.dat MD5-of-262144MD5-of-512CRC32C
0204c0baeeacbc4b5a3c8af5152944fe2d79
hdfs:///tmp/random-crctest2.dat MD5-of-262144MD5-of-512CRC32C
0204c0baeeacbc4b5a3c8af5152944fe2d79
hdfs:///tmp/random-crctest3.dat MD5-of-131072MD5-of-1024CRC32C
0402930b0d7ad333786a839b044ed8d18d2d
hdfs:///tmp/random-crctest4.dat MD5-of-131072MD5-of-512CRC32C
02028baa940ef6ed21fb4bd6224ce917d127
{code}
After:
{code:java}
$ hadoop fs -checksum hdfs:///tmp/random-crctest*.dat
hdfs:///tmp/random-crctest.dat COMPOSITE-CRC32C
4db86e2b
hdfs:///tmp/random-crctest2.dat COMPOSITE-CRC32C
4db86e2b
hdfs:///tmp/random-crctest3.dat COMPOSITE-CRC32C
4db86e2b
hdfs:///tmp/random-crctest4.dat COMPOSITE-CRC32C
4db86e2b
{code}
> Expose file-level composite CRCs in HDFS which are comparable across
> different instances/layouts
>
>
> Key: HDFS-13056
> URL: https://issues.apache.org/jira/browse/HDFS-13056
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, distcp, erasure-coding, federation, hdfs
>Affects Versions: 3.0.0
>Reporter: Dennis Huo
>Priority: Major
> Attachments: HDFS-13056-branch-2.8.poc1.patch,
> hdfs-file-composite-crc32-v1.pdf
>
>
> FileChecksum was first introduced in
> [https://issues-test.apache.org/jira/browse/HADOOP-3981] and ever since then
> has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are
> already stored as part of datanode metadata, and the MD5 approach is used to
> compute an aggregate value in a distributed manner, with individual datanodes
> computing the MD5-of-CRCs per-block in parallel, and the HDFS client
> computing the second-level MD5.
>
> A shortcoming of this approach which is often brought up is the fact that
> this FileChecksum is sensitive to the internal block-size and chunk-size
> configuration, and thus different HDFS files with different block/chunk
> settings cannot be compared. More commonly, one might have different HDFS
> clusters which use different block sizes, in which case any data migration
> won't be able to use the FileChecksum for distcp's rsync functionality or for
> verifying end-to-end data integrity (on top of low-level data integrity
> checks applied at data transfer time).
>
> This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430
> during the addition of checksum support for striped erasure-coded files;
> while there was some discussion of using CRC composability, it still
> ultimately settled on hierarchical MD5 approach, which also adds the problem
> that checksums of basic replicated files are not comparable to striped files.
>
> This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses
> CRC composition to remain completely chunk/block agnostic, and allows
> comparison between striped vs replicated files, between different HDFS
> instances, and possible even between HDFS and other external storage systems.
> This feature can also be added in-place to be compatible with existing block
> metadata, and doesn't need to change the normal path of chunk verification,
> so is minimally invasive. This also means even large preexisting HDFS
> deployments could adopt this feature to retroactively sync data. A detailed
> design
[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056-branch-2.8.poc1.patch > Expose file-level composite CRCs in HDFS which are comparable across > different instances/layouts > > > Key: HDFS-13056 > URL: https://issues.apache.org/jira/browse/HDFS-13056 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, erasure-coding, federation, hdfs >Affects Versions: 3.0.0 >Reporter: Dennis Huo >Priority: Major > Attachments: HDFS-13056-branch-2.8.poc1.patch, > hdfs-file-composite-crc32-v1.pdf > > > FileChecksum was first introduced in > [https://issues-test.apache.org/jira/browse/HADOOP-3981] and ever since then > has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are > already stored as part of datanode metadata, and the MD5 approach is used to > compute an aggregate value in a distributed manner, with individual datanodes > computing the MD5-of-CRCs per-block in parallel, and the HDFS client > computing the second-level MD5. > > A shortcoming of this approach which is often brought up is the fact that > this FileChecksum is sensitive to the internal block-size and chunk-size > configuration, and thus different HDFS files with different block/chunk > settings cannot be compared. More commonly, one might have different HDFS > clusters which use different block sizes, in which case any data migration > won't be able to use the FileChecksum for distcp's rsync functionality or for > verifying end-to-end data integrity (on top of low-level data integrity > checks applied at data transfer time). > > This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430 > during the addition of checksum support for striped erasure-coded files; > while there was some discussion of using CRC composability, it still > ultimately settled on hierarchical MD5 approach, which also adds the problem > that checksums of basic replicated files are not comparable to striped files. > > This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses > CRC composition to remain completely chunk/block agnostic, and allows > comparison between striped vs replicated files, between different HDFS > instances, and possible even between HDFS and other external storage systems. > This feature can also be added in-place to be compatible with existing block > metadata, and doesn't need to change the normal path of chunk verification, > so is minimally invasive. This also means even large preexisting HDFS > deployments could adopt this feature to retroactively sync data. A detailed > design document can be found here: > https://storage.googleapis.com/dennishuo/hdfs-file-composite-crc32-v1.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12772) RBF: Federation Router State State Store internal API
[ https://issues.apache.org/jira/browse/HDFS-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336840#comment-16336840 ] Hudson commented on HDFS-12772: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13543 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13543/]) HDFS-12772. RBF: Federation Router State State Store internal API. (inigoiri: rev 95743c672e6b42b227a22dfa7cc16edc7bdb58bb) * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/GetRouterRegistrationsResponse.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/records/impl/pb/StateStoreVersionPBImpl.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterServiceState.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/GetRouterRegistrationsRequest.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/FederationProtocol.proto * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/RouterHeartbeatRequest.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/records/RouterState.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/impl/RouterStoreImpl.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/RouterHeartbeatResponse.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/GetRouterRegistrationResponsePBImpl.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/GetRouterRegistrationsResponsePBImpl.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/RouterHeartbeatRequestPBImpl.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/RouterHeartbeatResponsePBImpl.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/GetRouterRegistrationRequestPBImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/store/driver/TestStateStoreDriverBase.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/FederationUtil.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/store/records/TestRouterState.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/GetRouterRegistrationsRequestPBImpl.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/GetRouterRegistrationResponse.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/records/StateStoreVersion.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/GetRouterRegistrationRequest.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/records/impl/pb/RouterStatePBImpl.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/RouterStore.java > RBF: Federation Router State State Store internal API > - > > Key: HDFS-12772 > URL: https://issues.apache.org/jira/browse/HDFS-12772 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1 > > Attachments: HDFS-12772.000.patch, HDFS-12772.001.patch, > HDFS-12772.002.patch, HDFS-12772.003.patch, HDFS-12772.004.patch, > HDFS-12772.005.patch > > > To monitor the state of the cluster, we should track the state of the > routers. This should be exposed in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12772) RBF: Federation Router State State Store internal API
[ https://issues.apache.org/jira/browse/HDFS-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336830#comment-16336830 ] Íñigo Goiri commented on HDFS-12772: Committed to trunk, branch-3.0, branch-2, and branch-2.9. Thanks for the review [~hanishakoneru] and [~linyiqun]. > RBF: Federation Router State State Store internal API > - > > Key: HDFS-12772 > URL: https://issues.apache.org/jira/browse/HDFS-12772 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1 > > Attachments: HDFS-12772.000.patch, HDFS-12772.001.patch, > HDFS-12772.002.patch, HDFS-12772.003.patch, HDFS-12772.004.patch, > HDFS-12772.005.patch > > > To monitor the state of the cluster, we should track the state of the > routers. This should be exposed in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12772) RBF: Federation Router State State Store internal API
[ https://issues.apache.org/jira/browse/HDFS-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-12772: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.1 2.9.1 2.10.0 3.1.0 Status: Resolved (was: Patch Available) > RBF: Federation Router State State Store internal API > - > > Key: HDFS-12772 > URL: https://issues.apache.org/jira/browse/HDFS-12772 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1 > > Attachments: HDFS-12772.000.patch, HDFS-12772.001.patch, > HDFS-12772.002.patch, HDFS-12772.003.patch, HDFS-12772.004.patch, > HDFS-12772.005.patch > > > To monitor the state of the cluster, we should track the state of the > routers. This should be exposed in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13042) RBF: Heartbeat Router State
[ https://issues.apache.org/jira/browse/HDFS-13042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13042: --- Attachment: HDFS-13042.001.patch > RBF: Heartbeat Router State > --- > > Key: HDFS-13042 > URL: https://issues.apache.org/jira/browse/HDFS-13042 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13042.000.patch, HDFS-13042.001.patch > > > The Router should heartbeat its state to the State Store. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12963) Error log level in ShortCircuitRegistry#removeShm
[
https://issues.apache.org/jira/browse/HDFS-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336825#comment-16336825
]
Hudson commented on HDFS-12963:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13542 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13542/])
HDFS-12963. Error log level in ShortCircuitRegistry#removeShm. (yqlin: rev
d95c13774e1bd5b3cc61bf4da8bae4a93ed0040c)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java
> Error log level in ShortCircuitRegistry#removeShm
> -
>
> Key: HDFS-12963
> URL: https://issues.apache.org/jira/browse/HDFS-12963
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hu xiaodong
>Assignee: hu xiaodong
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12963.001.patch
>
>
> {code:title=org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.java|borderStyle=solid}
> public synchronized void removeShm(ShortCircuitShm shm) {
> if (LOG.isTraceEnabled()) {
> LOG.debug("removing shm " + shm); -- I think here
> should be trace
> }
> // Stop tracking the shmId.
> RegisteredShm removedShm = segments.remove(shm.getShmId());
> Preconditions.checkState(removedShm == shm,
> "failed to remove " + shm.getShmId());
> // Stop tracking the slots.
> for (Iterator iter = shm.slotIterator(); iter.hasNext(); ) {
> Slot slot = iter.next();
> boolean removed = slots.remove(slot.getBlockId(), slot);
> Preconditions.checkState(removed);
> slot.makeInvalid();
> }
> // De-allocate the memory map and close the shared file.
> shm.free();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12963) Error log level in ShortCircuitRegistry#removeShm
[
https://issues.apache.org/jira/browse/HDFS-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336800#comment-16336800
]
hu xiaodong commented on HDFS-12963:
thank you, [~linyiqun]
> Error log level in ShortCircuitRegistry#removeShm
> -
>
> Key: HDFS-12963
> URL: https://issues.apache.org/jira/browse/HDFS-12963
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hu xiaodong
>Assignee: hu xiaodong
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12963.001.patch
>
>
> {code:title=org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.java|borderStyle=solid}
> public synchronized void removeShm(ShortCircuitShm shm) {
> if (LOG.isTraceEnabled()) {
> LOG.debug("removing shm " + shm); -- I think here
> should be trace
> }
> // Stop tracking the shmId.
> RegisteredShm removedShm = segments.remove(shm.getShmId());
> Preconditions.checkState(removedShm == shm,
> "failed to remove " + shm.getShmId());
> // Stop tracking the slots.
> for (Iterator iter = shm.slotIterator(); iter.hasNext(); ) {
> Slot slot = iter.next();
> boolean removed = slots.remove(slot.getBlockId(), slot);
> Preconditions.checkState(removed);
> slot.makeInvalid();
> }
> // De-allocate the memory map and close the shared file.
> shm.free();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12963) error log level in ShortCircuitRegistry#removeShm
[
https://issues.apache.org/jira/browse/HDFS-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12963:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
Target Version/s: 3.1.0 (was: 3.0.0-alpha4)
Status: Resolved (was: Patch Available)
Committed this to trunk. Thanks [~xiaodong.hu] for the contibution and thanks
[~shahrs87] for the review.
> error log level in ShortCircuitRegistry#removeShm
> -
>
> Key: HDFS-12963
> URL: https://issues.apache.org/jira/browse/HDFS-12963
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hu xiaodong
>Assignee: hu xiaodong
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12963.001.patch
>
>
> {code:title=org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.java|borderStyle=solid}
> public synchronized void removeShm(ShortCircuitShm shm) {
> if (LOG.isTraceEnabled()) {
> LOG.debug("removing shm " + shm); -- I think here
> should be trace
> }
> // Stop tracking the shmId.
> RegisteredShm removedShm = segments.remove(shm.getShmId());
> Preconditions.checkState(removedShm == shm,
> "failed to remove " + shm.getShmId());
> // Stop tracking the slots.
> for (Iterator iter = shm.slotIterator(); iter.hasNext(); ) {
> Slot slot = iter.next();
> boolean removed = slots.remove(slot.getBlockId(), slot);
> Preconditions.checkState(removed);
> slot.makeInvalid();
> }
> // De-allocate the memory map and close the shared file.
> shm.free();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12963) Error log level in ShortCircuitRegistry#removeShm
[
https://issues.apache.org/jira/browse/HDFS-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12963:
-
Summary: Error log level in ShortCircuitRegistry#removeShm (was: error log
level in ShortCircuitRegistry#removeShm)
> Error log level in ShortCircuitRegistry#removeShm
> -
>
> Key: HDFS-12963
> URL: https://issues.apache.org/jira/browse/HDFS-12963
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hu xiaodong
>Assignee: hu xiaodong
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12963.001.patch
>
>
> {code:title=org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.java|borderStyle=solid}
> public synchronized void removeShm(ShortCircuitShm shm) {
> if (LOG.isTraceEnabled()) {
> LOG.debug("removing shm " + shm); -- I think here
> should be trace
> }
> // Stop tracking the shmId.
> RegisteredShm removedShm = segments.remove(shm.getShmId());
> Preconditions.checkState(removedShm == shm,
> "failed to remove " + shm.getShmId());
> // Stop tracking the slots.
> for (Iterator iter = shm.slotIterator(); iter.hasNext(); ) {
> Slot slot = iter.next();
> boolean removed = slots.remove(slot.getBlockId(), slot);
> Preconditions.checkState(removed);
> slot.makeInvalid();
> }
> // De-allocate the memory map and close the shared file.
> shm.free();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12963) error log level in ShortCircuitRegistry#removeShm
[
https://issues.apache.org/jira/browse/HDFS-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336786#comment-16336786
]
Yiqun Lin commented on HDFS-12963:
--
+1, will commit this shortly.
> error log level in ShortCircuitRegistry#removeShm
> -
>
> Key: HDFS-12963
> URL: https://issues.apache.org/jira/browse/HDFS-12963
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hu xiaodong
>Assignee: hu xiaodong
>Priority: Minor
> Attachments: HDFS-12963.001.patch
>
>
> {code:title=org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.java|borderStyle=solid}
> public synchronized void removeShm(ShortCircuitShm shm) {
> if (LOG.isTraceEnabled()) {
> LOG.debug("removing shm " + shm); -- I think here
> should be trace
> }
> // Stop tracking the shmId.
> RegisteredShm removedShm = segments.remove(shm.getShmId());
> Preconditions.checkState(removedShm == shm,
> "failed to remove " + shm.getShmId());
> // Stop tracking the slots.
> for (Iterator iter = shm.slotIterator(); iter.hasNext(); ) {
> Slot slot = iter.next();
> boolean removed = slots.remove(slot.getBlockId(), slot);
> Preconditions.checkState(removed);
> slot.makeInvalid();
> }
> // De-allocate the memory map and close the shared file.
> shm.free();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336782#comment-16336782
]
Yongjun Zhang commented on HDFS-12051:
--
Hi [~mi...@cloudera.com],
Thanks for the updated patch, my apology I did not point out earlier:
For the config change, we need to modify
./hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml file
accordingly.
1. We need to specify that the old config is obsoleted and new config will be
used in the old config section. Please see examples in the same file like this.
for example
{code}
dfs.web.ugi
dfs.web.ugi is deprecated. Use hadoop.http.staticuser.user instead.
{code}
Suggest to describe there that the implementation is changed etc, as I
mentioned earlier.
2. We need to add a new section in this file for the new config, with some good
description what it means and how to use it.
Thanks.
> Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly
> those denoting file/directory names) to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch, HDFS-12051.07.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code:java}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <--
[jira] [Commented] (HDFS-12772) RBF: Federation Router State State Store internal API
[ https://issues.apache.org/jira/browse/HDFS-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336779#comment-16336779 ] Yiqun Lin commented on HDFS-12772: -- LGTM, +1. > RBF: Federation Router State State Store internal API > - > > Key: HDFS-12772 > URL: https://issues.apache.org/jira/browse/HDFS-12772 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-12772.000.patch, HDFS-12772.001.patch, > HDFS-12772.002.patch, HDFS-12772.003.patch, HDFS-12772.004.patch, > HDFS-12772.005.patch > > > To monitor the state of the cluster, we should track the state of the > routers. This should be exposed in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-12051:
-
Hadoop Flags: Incompatible change
> Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly
> those denoting file/directory names) to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch, HDFS-12051.07.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code:java}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
>
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[
https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDFS-13055:
--
Attachment: HDFS-13055.001.patch
> Aggregate usage statistics from datanodes
> -
>
> Key: HDFS-13055
> URL: https://issues.apache.org/jira/browse/HDFS-13055
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-13055.001.patch
>
>
> We collect variety of statistics in DataNodes and expose them via JMX.
> Aggregating some of the high level statistics which we are already collecting
> in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable
> time window will create a central repository accessible via JMX and UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13055) Aggregate usage statistics from datanodes
[
https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336760#comment-16336760
]
Ajay Kumar commented on HDFS-13055:
---
Attaching initial patch for review. Will add test case soon.
> Aggregate usage statistics from datanodes
> -
>
> Key: HDFS-13055
> URL: https://issues.apache.org/jira/browse/HDFS-13055
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-13055.001.patch
>
>
> We collect variety of statistics in DataNodes and expose them via JMX.
> Aggregating some of the high level statistics which we are already collecting
> in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable
> time window will create a central repository accessible via JMX and UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10419) Building HDFS on top of Ozone's storage containers
[ https://issues.apache.org/jira/browse/HDFS-10419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336759#comment-16336759 ] Konstantin Shvachko commented on HDFS-10419: Read the document, but did not get enough information about the architecture for Milestones 1 and 2. [~sanjay.radia] it would be good to add more details on the changes to the NameNode, when blocks are maintained on Ozone. I can partially deduce the ideas behind Milestone 1, but Milestone 2 is not clear to me. As a side note, I didn't understand why you used 50MB blocks in your math. The default is 128MB and many people run HDFS with 512MB blocks. > Building HDFS on top of Ozone's storage containers > -- > > Key: HDFS-10419 > URL: https://issues.apache.org/jira/browse/HDFS-10419 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Major > Attachments: Evolving NN using new block-container layer.pdf > > > In HDFS-7240, Ozone defines storage containers to store both the data and the > metadata. The storage container layer provides an object storage interface > and aims to manage data/metadata in a distributed manner. More details about > storage containers can be found in the design doc in HDFS-7240. > HDFS can adopt the storage containers to store and manage blocks. The general > idea is: > # Each block can be treated as an object and the block ID is the object's key. > # Blocks will still be stored in DataNodes but as objects in storage > containers. > # The block management work can be separated out of the NameNode and will be > handled by the storage container layer in a more distributed way. The > NameNode will only manage the namespace (i.e., files and directories). > # For each file, the NameNode only needs to record a list of block IDs which > are used as keys to obtain real data from storage containers. > # A new DFSClient implementation talks to both NameNode and the storage > container layer to read/write. > HDFS, especially the NameNode, can get much better scalability from this > design. Currently the NameNode's heaviest workload comes from the block > management, which includes maintaining the block-DataNode mapping, receiving > full/incremental block reports, tracking block states (under/over/miss > replicated), and joining every writing pipeline protocol to guarantee the > data consistency. These work bring high memory footprint and make NameNode > suffer from GC. HDFS-5477 already proposes to convert BlockManager as a > service. If we can build HDFS on top of the storage container layer, we not > only separate out the BlockManager from the NameNode, but also replace it > with a new distributed management scheme. > The storage container work is currently in progress in HDFS-7240, and the > work proposed here is still in an experimental/exploring stage. We can do > this experiment in a feature branch so that people with interests can be > involved. > A design doc will be uploaded later explaining more details. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[
https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDFS-13055:
--
Attachment: (was: HDFS-13055.001.patch)
> Aggregate usage statistics from datanodes
> -
>
> Key: HDFS-13055
> URL: https://issues.apache.org/jira/browse/HDFS-13055
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
>
> We collect variety of statistics in DataNodes and expose them via JMX.
> Aggregating some of the high level statistics which we are already collecting
> in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable
> time window will create a central repository accessible via JMX and UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[
https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDFS-13055:
--
Attachment: HDFS-13055.001.patch
> Aggregate usage statistics from datanodes
> -
>
> Key: HDFS-13055
> URL: https://issues.apache.org/jira/browse/HDFS-13055
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-13055.001.patch
>
>
> We collect variety of statistics in DataNodes and expose them via JMX.
> Aggregating some of the high level statistics which we are already collecting
> in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable
> time window will create a central repository accessible via JMX and UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12963) error log level in ShortCircuitRegistry#removeShm
[
https://issues.apache.org/jira/browse/HDFS-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336749#comment-16336749
]
hu xiaodong edited comment on HDFS-12963 at 1/24/18 2:00 AM:
-
hello, [~ajisakaa],can you commit this? thank you very much.
was (Author: xiaodong.hu):
hello, [~ajisakaa],can you commit this? than you very much.
> error log level in ShortCircuitRegistry#removeShm
> -
>
> Key: HDFS-12963
> URL: https://issues.apache.org/jira/browse/HDFS-12963
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hu xiaodong
>Assignee: hu xiaodong
>Priority: Minor
> Attachments: HDFS-12963.001.patch
>
>
> {code:title=org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.java|borderStyle=solid}
> public synchronized void removeShm(ShortCircuitShm shm) {
> if (LOG.isTraceEnabled()) {
> LOG.debug("removing shm " + shm); -- I think here
> should be trace
> }
> // Stop tracking the shmId.
> RegisteredShm removedShm = segments.remove(shm.getShmId());
> Preconditions.checkState(removedShm == shm,
> "failed to remove " + shm.getShmId());
> // Stop tracking the slots.
> for (Iterator iter = shm.slotIterator(); iter.hasNext(); ) {
> Slot slot = iter.next();
> boolean removed = slots.remove(slot.getBlockId(), slot);
> Preconditions.checkState(removed);
> slot.makeInvalid();
> }
> // De-allocate the memory map and close the shared file.
> shm.free();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12963) error log level in ShortCircuitRegistry#removeShm
[
https://issues.apache.org/jira/browse/HDFS-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336749#comment-16336749
]
hu xiaodong commented on HDFS-12963:
hello, [~ajisakaa],can you commit this? than you very much.
> error log level in ShortCircuitRegistry#removeShm
> -
>
> Key: HDFS-12963
> URL: https://issues.apache.org/jira/browse/HDFS-12963
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hu xiaodong
>Assignee: hu xiaodong
>Priority: Minor
> Attachments: HDFS-12963.001.patch
>
>
> {code:title=org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.java|borderStyle=solid}
> public synchronized void removeShm(ShortCircuitShm shm) {
> if (LOG.isTraceEnabled()) {
> LOG.debug("removing shm " + shm); -- I think here
> should be trace
> }
> // Stop tracking the shmId.
> RegisteredShm removedShm = segments.remove(shm.getShmId());
> Preconditions.checkState(removedShm == shm,
> "failed to remove " + shm.getShmId());
> // Stop tracking the slots.
> for (Iterator iter = shm.slotIterator(); iter.hasNext(); ) {
> Slot slot = iter.next();
> boolean removed = slots.remove(slot.getBlockId(), slot);
> Preconditions.checkState(removed);
> slot.makeInvalid();
> }
> // De-allocate the memory map and close the shared file.
> shm.free();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12522) Ozone: Remove the Priority Queues used in the Container State Manager
[
https://issues.apache.org/jira/browse/HDFS-12522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336723#comment-16336723
]
Anu Engineer commented on HDFS-12522:
-
[~xyao] and [~nandakumar131] thanks for the review comments. I have attached v2
patch that addresses all comments. The details are below:
{quote}Line:119 Instead of storing ContainerInfo in lastUsedMap we can just
store ContainerID.
{quote}
Fixed.
{quote}Line:310 Do we need handle Integer overflow case for containerCount?
{quote}
Fixed by adding a precondition check in setContainerID()
{quote}Line:313 If containers.createContainer(containerInfo) call fails, should
we revert containerCount?
{quote}
I have bypassed this problem completely, by making the container ID a 64 bit
value for time being. That means we don't have to worry about leaking a
container ID. We have enough of them.
{quote}Line:354 Javadoc to be updated stating that getMatchingContainer will
return null if there are no matching containers available.
{quote}
Fixed.
{quote}Line:363 containers.getMatchingContainerIDs can return null, so if
(matchingSet.size() == 0) has to be if (matchingSet == null ||
matchingSet.size() == 0)
{quote}
Fixed.
{quote}Line:369 We should store lastUsedContainer for each combination of
owner, type and factor instead of just owner
{quote}
Fixed.
{quote}Line:424 Can we rename getMatchingContainers to getMatchingContainerIDs
as it returns NavigableSet?
{quote}
Fixed.
{{ContainerStateMap.java}}
{quote}Line:61 Typo Contianer -> Container
{quote}
Fixed.
{quote}Line:85 emptySet can be static.
{quote}
Fixed.
{quote}Line:115 rename createContainer to addContainer, as it does not create
any container but adds them to ContainerStateMap?
{quote}
{quote}Line:118, 162, 183, 234, 248, 262, 299 lock instance doesn't need to be
assigned to a variable.
{quote}
I know in Java 9, we don't need to do this, but my complier warns if I try to
remove this Java 8.
Line:141 Do we need getCurrentInfo(ContainerInfo info)? we can always use
getContainerInfo(int containerID).
Line:151 getContainerInfo can directly take ContainerID object instead of
creating new ContainerID object for each call (unnecessary object creation)
Both of these helper functions with appropriate uses. I have renamed
getCurentInfo to getContianerInfo.
{quote}Line:200 We should not use ContainerInfo that is passed as argument, it
will contain stale allocatedBytes value as it’s read from metadata db
(allocatedBytes value is updated in db only during SCM shutdown). We should
always do containerMap.get, update the state and then containerMap.put
{quote}
That does *not* work. Since we don't know what values of info is being updated.
If we want allocatedBytes to be corect, then it is has to updated at a level
higher than this.
updateState: We don’t need ContainerInfo as an argument, we can just have
containerID
Technically yes, but unfortunately, we use this to update all fields of the
ContainerInfo. That is when state change is executed, the user can change other
fields too.
{quote}Rename suggestions:
{quote}
Fixed.
{quote}Line:354 - 355 Random “+” & “*” in the javadoc.
{quote}
Fixed.
{quote}Line: 443 - 444 This can be removed.
{quote}
Line numbers are not correct?
{quote}Since we have removed PriorityQueue from ContainerStateManager, we don’t
need lastUsed time in ContainerInfo. We will be maintaining that information in
ContainerStateManager#lastUsedMap.
{quote}
We can also remove ContainerInfo#compare and ContainerInfo#compareTo methods.
I think this is still good metric to have, especially for debugging and testing
scenerios. We can assert that all containers have been used in the last x mins
etc. Not removing this for now. We can always revisit this. I think this is a
good stat to have in container info.
{quote}setState : Having setter method in ContainerInfo, this one is just a
concern which was raised in [1] and [2] comment.
{quote}
We will have to have a client facing class which is different from the one we
use in the server. We will have to fix that in a different JIRA. I am going to
file or update one of the related JIRAs.
{quote}This class is not referred/used anywhere, is this required? Am I missing
something?
{quote}
Started using this class.
{quote}Line:131 We are not using final long cacheSize anywhere, can be removed
from the constructor
{quote}
Fixed.
{quote}Line:433 The TODO can be removed, it has been fixed in HDFS-12751
{quote}
Fixed.
{{TestContainerStateManager.java}} – Not able to correlate the comments to the
line number in comments.
{quote}Line 52: unused imports.
{quote}
Fixed.
{quote}Line 118: can we use FAILED_TO_CHANGE_CONTAINER_STATE without adding
FAILED_TO_UPDATE_CONTAINER_STATE?
{quote}
Fixed.
{quote}Line 119: CONTAINER_MAPS_NO_SUCH_STATE is not used anywhere, can we
remove it?
{quote}
Fixed.
{quote}Line 120: can we use CONTAINER_EXISTS without adding
[jira] [Updated] (HDFS-12522) Ozone: Remove the Priority Queues used in the Container State Manager
[ https://issues.apache.org/jira/browse/HDFS-12522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12522: Attachment: HDFS-12522-HDFS-7240.002.patch > Ozone: Remove the Priority Queues used in the Container State Manager > - > > Key: HDFS-12522 > URL: https://issues.apache.org/jira/browse/HDFS-12522 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > Attachments: HDFS-12522-HDFS-7240.001.patch, > HDFS-12522-HDFS-7240.002.patch > > > During code review of HDFS-12387, it was suggested that we remove the > priority queues that was used in ContainerStateManager. This JIRA tracks that > issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: hdfs-file-composite-crc32-v1.pdf > Expose file-level composite CRCs in HDFS which are comparable across > different instances/layouts > > > Key: HDFS-13056 > URL: https://issues.apache.org/jira/browse/HDFS-13056 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, erasure-coding, federation, hdfs >Affects Versions: 3.0.0 >Reporter: Dennis Huo >Priority: Major > Attachments: hdfs-file-composite-crc32-v1.pdf > > > FileChecksum was first introduced in > [https://issues-test.apache.org/jira/browse/HADOOP-3981] and ever since then > has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are > already stored as part of datanode metadata, and the MD5 approach is used to > compute an aggregate value in a distributed manner, with individual datanodes > computing the MD5-of-CRCs per-block in parallel, and the HDFS client > computing the second-level MD5. > > A shortcoming of this approach which is often brought up is the fact that > this FileChecksum is sensitive to the internal block-size and chunk-size > configuration, and thus different HDFS files with different block/chunk > settings cannot be compared. More commonly, one might have different HDFS > clusters which use different block sizes, in which case any data migration > won't be able to use the FileChecksum for distcp's rsync functionality or for > verifying end-to-end data integrity (on top of low-level data integrity > checks applied at data transfer time). > > This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430 > during the addition of checksum support for striped erasure-coded files; > while there was some discussion of using CRC composability, it still > ultimately settled on hierarchical MD5 approach, which also adds the problem > that checksums of basic replicated files are not comparable to striped files. > > This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses > CRC composition to remain completely chunk/block agnostic, and allows > comparison between striped vs replicated files, between different HDFS > instances, and possible even between HDFS and other external storage systems. > This feature can also be added in-place to be compatible with existing block > metadata, and doesn't need to change the normal path of chunk verification, > so is minimally invasive. This also means even large preexisting HDFS > deployments could adopt this feature to retroactively sync data. A detailed > design document can be found here: > https://storage.googleapis.com/dennishuo/hdfs-file-composite-crc32-v1.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
Dennis Huo created HDFS-13056: - Summary: Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts Key: HDFS-13056 URL: https://issues.apache.org/jira/browse/HDFS-13056 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, distcp, erasure-coding, federation, hdfs Affects Versions: 3.0.0 Reporter: Dennis Huo FileChecksum was first introduced in [https://issues-test.apache.org/jira/browse/HADOOP-3981] and ever since then has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are already stored as part of datanode metadata, and the MD5 approach is used to compute an aggregate value in a distributed manner, with individual datanodes computing the MD5-of-CRCs per-block in parallel, and the HDFS client computing the second-level MD5. A shortcoming of this approach which is often brought up is the fact that this FileChecksum is sensitive to the internal block-size and chunk-size configuration, and thus different HDFS files with different block/chunk settings cannot be compared. More commonly, one might have different HDFS clusters which use different block sizes, in which case any data migration won't be able to use the FileChecksum for distcp's rsync functionality or for verifying end-to-end data integrity (on top of low-level data integrity checks applied at data transfer time). This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430 during the addition of checksum support for striped erasure-coded files; while there was some discussion of using CRC composability, it still ultimately settled on hierarchical MD5 approach, which also adds the problem that checksums of basic replicated files are not comparable to striped files. This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses CRC composition to remain completely chunk/block agnostic, and allows comparison between striped vs replicated files, between different HDFS instances, and possible even between HDFS and other external storage systems. This feature can also be added in-place to be compatible with existing block metadata, and doesn't need to change the normal path of chunk verification, so is minimally invasive. This also means even large preexisting HDFS deployments could adopt this feature to retroactively sync data. A detailed design document can be found here: https://storage.googleapis.com/dennishuo/hdfs-file-composite-crc32-v1.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13018) Block Storage: make the iscsi target addres configurable for discovery
[
https://issues.apache.org/jira/browse/HDFS-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336641#comment-16336641
]
genericqa commented on HDFS-13018:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
5s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 50s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
58s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}134m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}213m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeysRatis |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.ozone.TestStorageContainerManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
| JIRA Issue | HDFS-13018 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907356/HDFS-13018-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 9a7708a9830e
[jira] [Commented] (HDFS-13042) RBF: Heartbeat Router State
[
https://issues.apache.org/jira/browse/HDFS-13042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336635#comment-16336635
]
genericqa commented on HDFS-13042:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
59s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 49s{color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs generated 19 new + 375 unchanged
- 19 fixed = 394 total (was 394) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 7 new + 431 unchanged - 0 fixed = 438 total (was 431) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 58s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestPread |
| | hadoop.hdfs.server.namenode.TestAddStripedBlocks |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy |
| | hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| | hadoop.hdfs.server.namenode.TestEditLog |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.server.namenode.TestReencryption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13042 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907328/HDFS-13042.000.patch |
| Optional Tests |
[jira] [Commented] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336622#comment-16336622
]
Uma Maheswara Rao G commented on HDFS-13033:
+1 on the latest patch.
Test failures unrelated and checkstyle comment is wrongly reporting about else
if and indentation. Other minor checkstyle issue about formatting was corrected
while committing.
Committed to branch.
> [SPS]: Implement a mechanism to do file block movements for external SPS
>
>
> Key: HDFS-13033
> URL: https://issues.apache.org/jira/browse/HDFS-13033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Fix For: HDFS-10285
>
> Attachments: HDFS-13033-HDFS-10285-00.patch,
> HDFS-13033-HDFS-10285-01.patch, HDFS-13033-HDFS-10285-02.patch,
> HDFS-13033-HDFS-10285-03.patch, HDFS-13033-HDFS-10285-04.patch
>
>
> HDFS-12911 modularization is introducing \{{BlockMoveTaskHandler}} interface
> for moving the file blocks. That will help us to plugin different ways of
> block move mechanisms, if needed.
> For Internal SPS, we have simple blk movement tasks to target DN descriptors.
> For external SPS, we should have mechanism to send \{{replaceBlock}} on
> target node and have a listener to track the block movement completion.
> This is the task to implement the \{{ExternalSPSBlockMoveTaskHandler}} plugin
> for external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-13033:
---
Resolution: Fixed
Fix Version/s: HDFS-10285
Status: Resolved (was: Patch Available)
> [SPS]: Implement a mechanism to do file block movements for external SPS
>
>
> Key: HDFS-13033
> URL: https://issues.apache.org/jira/browse/HDFS-13033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Fix For: HDFS-10285
>
> Attachments: HDFS-13033-HDFS-10285-00.patch,
> HDFS-13033-HDFS-10285-01.patch, HDFS-13033-HDFS-10285-02.patch,
> HDFS-13033-HDFS-10285-03.patch, HDFS-13033-HDFS-10285-04.patch
>
>
> HDFS-12911 modularization is introducing \{{BlockMoveTaskHandler}} interface
> for moving the file blocks. That will help us to plugin different ways of
> block move mechanisms, if needed.
> For Internal SPS, we have simple blk movement tasks to target DN descriptors.
> For external SPS, we should have mechanism to send \{{replaceBlock}} on
> target node and have a listener to track the block movement completion.
> This is the task to implement the \{{ExternalSPSBlockMoveTaskHandler}} plugin
> for external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13054) Handling PathIsNotEmptyDirectoryException in DFSClient delete call
[
https://issues.apache.org/jira/browse/HDFS-13054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336585#comment-16336585
]
genericqa commented on HDFS-13054:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
103 unchanged - 0 fixed = 104 total (was 103) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 8s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
19s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}154m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}207m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.server.federation.router.TestRouterRpc |
| | hadoop.hdfs.TestBlockMissingException |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestParallelUnixDomainRead |
[jira] [Commented] (HDFS-13054) Handling PathIsNotEmptyDirectoryException in DFSClient delete call
[
https://issues.apache.org/jira/browse/HDFS-13054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336582#comment-16336582
]
genericqa commented on HDFS-13054:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
103 unchanged - 0 fixed = 104 total (was 103) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 15s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
25s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}142m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}196m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.hdfs.TestFileAppend2 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
| | hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer |
| | hadoop.hdfs.TestDecommissionWithStriped |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestErasureCodingPolicies |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter |
| | hadoop.hdfs.TestFileCreationEmpty |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| |
[jira] [Commented] (HDFS-8921) Add an option to Balancer for specifying the k-most over-utilized DNs or all over-utilized DNs as sources.
[ https://issues.apache.org/jira/browse/HDFS-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336577#comment-16336577 ] Konstantin Shvachko commented on HDFS-8921: --- What's the status here? > Add an option to Balancer for specifying the k-most over-utilized DNs or all > over-utilized DNs as sources. > -- > > Key: HDFS-8921 > URL: https://issues.apache.org/jira/browse/HDFS-8921 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Plamen Jeliazkov >Priority: Major > Attachments: HDFS-8921.patch > > > Arpit suggested to add a separate option to source from the most > over-utilized DataNodes first so the administrator does not have to pass the > source DNs manually; see [this > comment|https://issues.apache.org/jira/browse/HDFS-8826?focusedCommentId=14700576=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14700576]. > The new option could allow specifying the k-most over-utilized DNs or all > over-utilized DNs as sources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13054) Handling PathIsNotEmptyDirectoryException in DFSClient delete call
[
https://issues.apache.org/jira/browse/HDFS-13054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336557#comment-16336557
]
genericqa commented on HDFS-13054:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
103 unchanged - 0 fixed = 104 total (was 103) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
42s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}133m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}198m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
| | hadoop.hdfs.tools.TestDFSAdminWithHA |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.web.TestWebHDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13054 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907344/HDFS-13054.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 43ec6333c971 3.13.0-135-generic #184-Ubuntu SMP
[jira] [Commented] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336551#comment-16336551
]
genericqa commented on HDFS-12051:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 1235 unchanged - 18 fixed = 1237 total (was 1253) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
8s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 53m 53s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
19s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Increment of volatile field
org.apache.hadoop.hdfs.server.namenode.NameCache.size in
org.apache.hadoop.hdfs.server.namenode.NameCache.put(byte[]) At
NameCache.java:in org.apache.hadoop.hdfs.server.namenode.NameCache.put(byte[])
At NameCache.java:[line 116] |
| Failed junit tests | hadoop.hdfs.TestModTime |
| | hadoop.hdfs.TestDataTransferKeepalive |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 |
| | hadoop.hdfs.TestReadStripedFileWithDecoding |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestHdfsAdmin |
| | hadoop.cli.TestHDFSCLI |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.hdfs.TestDFSStorageStateRecovery |
| | hadoop.cli.TestErasureCodingCLI |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.security.TestPermissionSymlinks |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.security.TestPermission |
| |
[jira] [Commented] (HDFS-13017) Block Storage: implement simple iscsi discovery in jscsi server
[
https://issues.apache.org/jira/browse/HDFS-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336540#comment-16336540
]
genericqa commented on HDFS-13017:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}177m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
| JIRA Issue | HDFS-13017 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907347/HDFS-13017-HDFS-7240.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 8073655bbe20 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-7240 / 65b9038 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22778/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test
[jira] [Commented] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336537#comment-16336537
]
genericqa commented on HDFS-13033:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
3s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
56s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-10285 has 1
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} HDFS-10285 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 177 unchanged - 0 fixed = 180 total (was 177) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}131m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}185m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.federation.metrics.TestFederationMetrics |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.federation.router.TestRouterRpc |
| | hadoop.hdfs.qjournal.TestSecureNNWithQJM |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.qjournal.server.TestJournalNode |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.TestReadStripedFileWithDNFailure |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13033 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907345/HDFS-13033-HDFS-10285-04.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 64e738968edd 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64
[jira] [Created] (HDFS-13055) Aggregate usage statistics from datanodes
Ajay Kumar created HDFS-13055:
-
Summary: Aggregate usage statistics from datanodes
Key: HDFS-13055
URL: https://issues.apache.org/jira/browse/HDFS-13055
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ajay Kumar
Assignee: Ajay Kumar
We collect variety of statistics in DataNodes and expose them via JMX.
Aggregating some of the high level statistics which we are already collecting
in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable
time window will create a central repository accessible via JMX and UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13042) RBF: Heartbeat Router State
[ https://issues.apache.org/jira/browse/HDFS-13042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13042: --- Status: Patch Available (was: Open) > RBF: Heartbeat Router State > --- > > Key: HDFS-13042 > URL: https://issues.apache.org/jira/browse/HDFS-13042 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13042.000.patch > > > The Router should heartbeat its state to the State Store. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12051:
--
Status: Patch Available (was: In Progress)
> Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly
> those denoting file/directory names) to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch, HDFS-12051.07.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code:java}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
[jira] [Commented] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336416#comment-16336416
]
Misha Dmitriev commented on HDFS-12051:
---
Thank you [~yzhangal], I've addressed your comment and uploaded the new patch.
> Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly
> those denoting file/directory names) to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch, HDFS-12051.07.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code:java}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
>
[jira] [Updated] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12051:
--
Status: In Progress (was: Patch Available)
> Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly
> those denoting file/directory names) to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch, HDFS-12051.07.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code:java}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
[jira] [Updated] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12051:
--
Attachment: HDFS-12051.07.patch
> Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly
> those denoting file/directory names) to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch, HDFS-12051.07.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code:java}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
>
[jira] [Commented] (HDFS-12772) RBF: Federation Router State State Store internal API
[ https://issues.apache.org/jira/browse/HDFS-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336408#comment-16336408 ] Íñigo Goiri commented on HDFS-12772: The unit tests are unrelated and the javac issue seems unrelated. I think this is good to go but I'll wait for proper approval to double check. > RBF: Federation Router State State Store internal API > - > > Key: HDFS-12772 > URL: https://issues.apache.org/jira/browse/HDFS-12772 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-12772.000.patch, HDFS-12772.001.patch, > HDFS-12772.002.patch, HDFS-12772.003.patch, HDFS-12772.004.patch, > HDFS-12772.005.patch > > > To monitor the state of the cluster, we should track the state of the > routers. This should be exposed in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13053) Track time to process packet in Datanode
[ https://issues.apache.org/jira/browse/HDFS-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336407#comment-16336407 ] Íñigo Goiri commented on HDFS-13053: Thanks [~hanishakoneru], it looks like HDFS-10917 pretty much duplicates this. [~pulkitmisra] is there anything not covered by that (or HDFS-11194)? > Track time to process packet in Datanode > > > Key: HDFS-13053 > URL: https://issues.apache.org/jira/browse/HDFS-13053 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Íñigo Goiri >Assignee: Pulkit Misra >Priority: Minor > Attachments: HDFS-13053.000.patch > > > We should track the time that each datanode takes to process a packet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336383#comment-16336383
]
Yongjun Zhang edited comment on HDFS-12051 at 1/23/18 9:32 PM:
---
Thanks for your comments [~szetszwo]. I thought you had reviewed the patch.
Misha described his tests at:
https://issues.apache.org/jira/browse/HDFS-12051?focusedCommentId=16084471=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16084471
and
https://issues.apache.org/jira/browse/HDFS-12051?focusedCommentId=16329891=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16329891
I agree more testing is always a good thing.
I did another round of review and found one thing I did not point out earlier
(I'm sorry about that). The patch obsoleted the old config
{code}
public static final String DFS_NAMENODE_NAME_CACHE_THRESHOLD_KEY =
"dfs.namenode.name.cache.threshold";
public static final int DFS_NAMENODE_NAME_CACHE_THRESHOLD_DEFAULT = 10;
{code}
and introduced new config
{code}
public static final String DFS_NAMENODE_NAME_CACHE_SIZE_KEY =
"dfs.namenode.name.cache.size";
public static final int DFS_NAMENODE_NAME_CACHE_SIZE_DEFAULT = 4 * 1024 *
1024;
{code}
We should make this visible to the user saying that the former is deprecated
because of the implementation change, and the new one can be used to config the
new implementation.
Would you please address this [~mi...@cloudera.com]?
Since I'm the only one who has reviewed the patch, I'm inviting [~manojg] for a
review (thanks Manoj). While he is reviewing, other folks are welcome to review.
Thanks.
was (Author: yzhangal):
Thanks for your comments [~szetszwo]. I thought you had reviewed the patch.
Misha described his tests at:
https://issues.apache.org/jira/browse/HDFS-12051?focusedCommentId=16084471=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16084471
and
https://issues.apache.org/jira/browse/HDFS-12051?focusedCommentId=16329891=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16329891
I agree more testing is always a good thing.
I did another round of review and found one thing I did not point out earlier.
The patch obsoleted the old config
{code}
public static final String DFS_NAMENODE_NAME_CACHE_THRESHOLD_KEY =
"dfs.namenode.name.cache.threshold";
public static final int DFS_NAMENODE_NAME_CACHE_THRESHOLD_DEFAULT = 10;
{code}
and introduced new config
{code}
public static final String DFS_NAMENODE_NAME_CACHE_SIZE_KEY =
"dfs.namenode.name.cache.size";
public static final int DFS_NAMENODE_NAME_CACHE_SIZE_DEFAULT = 4 * 1024 *
1024;
{code}
We should make this visible to the user saying that the former is deprecated
because of the implementation change, and the new one can be used to config the
new implementation.
Would you please address this [~mi...@cloudera.com]?
Since I'm the only one who has reviewed the patch, I'm inviting [~manojg] for a
review (thanks Manoj). While he is reviewing, other folks are welcome to review.
Thanks.
> Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly
> those denoting file/directory names) to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code:java}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of
[jira] [Commented] (HDFS-12654) APPEND API call is different in HTTPFS and NameNode REST
[
https://issues.apache.org/jira/browse/HDFS-12654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336386#comment-16336386
]
Andras Czesznak commented on HDFS-12654:
Hello [~iwasakims] San, the issue occurred with the FluentD app and described
here:
https://github.com/fluent/fluent-plugin-webhdfs/issues/46
>From the HTTPFS's log:
```
2017-10-03 16:20:59,204 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as: (auth:PROXY) via httpfs (auth:SIMPLE)
cause:java.io.FileNotFoundException: failed to append to non-existent file
/fluentd/process/.log for client
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:2930)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:3227)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:3191)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:614)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.append(AuthorizationProviderProxyClientProtocol.java:126)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:416)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
```
This is what it looks like in FluentD log
```
2017-10-04 17:23:54 -0700 [warn]: failed to communicate hdfs cluster, path:
/fluentd/process/.log
2017-10-04 17:23:54 -0700 [warn]: temporarily failed to flush the buffer.
next_retry=2017-10-04 17:24:24 -0700 error_class="WebHDFS::ServerError"
error="Failed to connect to host :14000, Broken pipe - sendfile"
plugin_id="object:3f8778538560"
2017-10-04 17:23:54 -0700 [warn]: suppressed same stacktrace
The source code snippets:
1) HTTPFS: direct FS output stream creation
/hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java
...
551 /**
552 * Append to an existing file (optional operation).
553 *
554 * IMPORTANT: The Progressable parameter is not used.
555 *
556 * @param f the existing file to be appended.
557 * @param bufferSize the size of the buffer to be used.
558 * @param progress for reporting progress if it is not null.
559 *
560 * @throws IOException
561 */
562 @Override
563 public FSDataOutputStream append(Path f, int bufferSize,
564 Progressable progress) throws IOException {
565 Map params = new HashMap();
566 params.put(OP_PARAM, Operation.APPEND.toString());
567 return uploadData(Operation.APPEND.getMethod(), f, params, bufferSize,
568 HttpURLConnection.HTTP_OK);
569 }
...
2) WebHDFS: indirect FS output stream creation thru DFS
...
/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
...
54 import org.apache.hadoop.fs.FSDataOutputStream;
...
795 /**
796 * Handle create/append output streams
797 */
798 class FsPathOutputStreamRunner extends
AbstractFsPathRunner {
799 private final int bufferSize;
800
801 FsPathOutputStreamRunner(Op op, Path fspath, int bufferSize,
802 Param... parameters) {
803 super(op, fspath, parameters);
804 this.bufferSize = bufferSize;
805 }
806
807 @Override
808 FSDataOutputStream getResponse(final HttpURLConnection conn)
809 throws IOException {
810 return new FSDataOutputStream(new BufferedOutputStream(
811 conn.getOutputStream(), bufferSize), statistics) {
812 @Override
813 public void close() throws IOException {
814 try {
815 super.close();
816 } finally {
817 try {
818 validateResponse(op, conn, true);
819 } finally {
820 conn.disconnect();
821 }
822 }
823 }
824 };
825 }
826 }
...
> APPEND API call is different in HTTPFS and NameNode REST
>
>
> Key: HDFS-12654
> URL: https://issues.apache.org/jira/browse/HDFS-12654
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, httpfs, namenode
>Affects Versions: 2.6.0, 2.7.0, 2.8.0, 3.0.0-beta1
>Reporter: Andras Czesznak
>Priority: Major
>
> The APPEND REST API call behaves differently in the NameNode REST and the
> HTTPFS codes. The
[jira] [Commented] (HDFS-12051) Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly those denoting file/directory names) to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336383#comment-16336383
]
Yongjun Zhang commented on HDFS-12051:
--
Thanks for your comments [~szetszwo]. I thought you had reviewed the patch.
Misha described his tests at:
https://issues.apache.org/jira/browse/HDFS-12051?focusedCommentId=16084471=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16084471
and
https://issues.apache.org/jira/browse/HDFS-12051?focusedCommentId=16329891=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16329891
I agree more testing is always a good thing.
I did another round of review and found one thing I did not point out earlier.
The patch obsoleted the old config
{code}
public static final String DFS_NAMENODE_NAME_CACHE_THRESHOLD_KEY =
"dfs.namenode.name.cache.threshold";
public static final int DFS_NAMENODE_NAME_CACHE_THRESHOLD_DEFAULT = 10;
{code}
and introduced new config
{code}
public static final String DFS_NAMENODE_NAME_CACHE_SIZE_KEY =
"dfs.namenode.name.cache.size";
public static final int DFS_NAMENODE_NAME_CACHE_SIZE_DEFAULT = 4 * 1024 *
1024;
{code}
We should make this visible to the user saying that the former is deprecated
because of the implementation change, and the new one can be used to config the
new implementation.
Would you please address this [~mi...@cloudera.com]?
Since I'm the only one who has reviewed the patch, I'm inviting [~manojg] for a
review (thanks Manoj). While he is reviewing, other folks are welcome to review.
Thanks.
> Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly
> those denoting file/directory names) to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code:java}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49,
[jira] [Commented] (HDFS-13018) Block Storage: make the iscsi target addres configurable for discovery
[ https://issues.apache.org/jira/browse/HDFS-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336358#comment-16336358 ] Elek, Marton commented on HDFS-13018: - Thanks to [Sebastian Graf|http://example.com/], the jscsi project is updated. Now we can use the new configuration values. I uploaded the patch. To test it: * Create a cblock custer, or you can use the docker-compose file from HDFS-12983 * Set the new configuration (in case of compose file add OZONE-SITE.XML_dfs.cblock.iscsi.advertised.port=3261 to the docker-config) * Create a cblock (docker-compose exec cblock hdfs cblock -c bilbo volume2 1GB 4) * Check the discovery: sudo iscsiadm -m discovery -t st -p localhost:3260 * The result should contain the modified port: 0.0.0.0:3261,1 bilbo:volume2 > Block Storage: make the iscsi target addres configurable for discovery > -- > > Key: HDFS-13018 > URL: https://issues.apache.org/jira/browse/HDFS-13018 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HDFS-13018-HDFS-7240.001.patch, > HDFS-13018-HDFS-7240.002.patch > > > Current jscsi server returns with the targetAddress (as ip) and 3260 (as > port) during the iscsi discovery. But in some cases we need to configure > these values. > For example in kubernetes the iscsi server could run behind a service where > the address (where the jscsi server is available from the cluster) could be > different from the targetAddress where the server is listening. > I propose to add two more configuration key to override the default > address/port for configuration but it also requires modification in the jscsi > project. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13018) Block Storage: make the iscsi target addres configurable for discovery
[ https://issues.apache.org/jira/browse/HDFS-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-13018: Status: Patch Available (was: Open) > Block Storage: make the iscsi target addres configurable for discovery > -- > > Key: HDFS-13018 > URL: https://issues.apache.org/jira/browse/HDFS-13018 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HDFS-13018-HDFS-7240.001.patch, > HDFS-13018-HDFS-7240.002.patch > > > Current jscsi server returns with the targetAddress (as ip) and 3260 (as > port) during the iscsi discovery. But in some cases we need to configure > these values. > For example in kubernetes the iscsi server could run behind a service where > the address (where the jscsi server is available from the cluster) could be > different from the targetAddress where the server is listening. > I propose to add two more configuration key to override the default > address/port for configuration but it also requires modification in the jscsi > project. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13018) Block Storage: make the iscsi target addres configurable for discovery
[ https://issues.apache.org/jira/browse/HDFS-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-13018: Attachment: HDFS-13018-HDFS-7240.002.patch > Block Storage: make the iscsi target addres configurable for discovery > -- > > Key: HDFS-13018 > URL: https://issues.apache.org/jira/browse/HDFS-13018 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HDFS-13018-HDFS-7240.001.patch, > HDFS-13018-HDFS-7240.002.patch > > > Current jscsi server returns with the targetAddress (as ip) and 3260 (as > port) during the iscsi discovery. But in some cases we need to configure > these values. > For example in kubernetes the iscsi server could run behind a service where > the address (where the jscsi server is available from the cluster) could be > different from the targetAddress where the server is listening. > I propose to add two more configuration key to override the default > address/port for configuration but it also requires modification in the jscsi > project. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13054) Handling PathIsNotEmptyDirectoryException in DFSClient delete call
[
https://issues.apache.org/jira/browse/HDFS-13054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336350#comment-16336350
]
Arpit Agarwal commented on HDFS-13054:
--
Thanks [~nandakumar131].
+1 pending Jenkins run.
> Handling PathIsNotEmptyDirectoryException in DFSClient delete call
> --
>
> Key: HDFS-13054
> URL: https://issues.apache.org/jira/browse/HDFS-13054
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13054.000.patch, HDFS-13054.001.patch,
> HDFS-13054.002.patch
>
>
> In {{DFSClient#delete}} call, if we get
> {{RemoteException(PathIsNotEmptyDirectoryException)}} we should unwrap and
> throw {{PathIsNotEmptyDirectoryException}} to the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13054) Handling PathIsNotEmptyDirectoryException in DFSClient delete call
[
https://issues.apache.org/jira/browse/HDFS-13054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336347#comment-16336347
]
Nanda kumar commented on HDFS-13054:
Thanks for the suggestion, yeah it is good to have Assert if the delete doesn't
throw any Exception.
Added the same in patch v002.
> Handling PathIsNotEmptyDirectoryException in DFSClient delete call
> --
>
> Key: HDFS-13054
> URL: https://issues.apache.org/jira/browse/HDFS-13054
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13054.000.patch, HDFS-13054.001.patch,
> HDFS-13054.002.patch
>
>
> In {{DFSClient#delete}} call, if we get
> {{RemoteException(PathIsNotEmptyDirectoryException)}} we should unwrap and
> throw {{PathIsNotEmptyDirectoryException}} to the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13053) Track time to process packet in Datanode
[
https://issues.apache.org/jira/browse/HDFS-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336346#comment-16336346
]
genericqa commented on HDFS-13053:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
5s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
58s{color} | {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 58s{color} |
{color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 58s{color}
| {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 53s{color} | {color:orange} hadoop-hdfs-project: The patch generated 10 new
+ 612 unchanged - 1 fixed = 622 total (was 613) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
35s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m
7s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
29s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13053 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907339/HDFS-13053.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux ce05d9960021 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a72cdcc |
|
[jira] [Updated] (HDFS-13054) Handling PathIsNotEmptyDirectoryException in DFSClient delete call
[
https://issues.apache.org/jira/browse/HDFS-13054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar updated HDFS-13054:
---
Attachment: HDFS-13054.002.patch
> Handling PathIsNotEmptyDirectoryException in DFSClient delete call
> --
>
> Key: HDFS-13054
> URL: https://issues.apache.org/jira/browse/HDFS-13054
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13054.000.patch, HDFS-13054.001.patch,
> HDFS-13054.002.patch
>
>
> In {{DFSClient#delete}} call, if we get
> {{RemoteException(PathIsNotEmptyDirectoryException)}} we should unwrap and
> throw {{PathIsNotEmptyDirectoryException}} to the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13054) Handling PathIsNotEmptyDirectoryException in DFSClient delete call
[
https://issues.apache.org/jira/browse/HDFS-13054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336335#comment-16336335
]
Arpit Agarwal commented on HDFS-13054:
--
One more suggestion, you can add a fail(...) call immediately after the
fs.delete call in the test case.
{code}
fs.delete(new Path("/tmp/nonEmptyDir"), false);
<>
{code}
So if no exception is thrown then the test case will fail.
+1 on the v1 patch either way, pending Jenkins.
> Handling PathIsNotEmptyDirectoryException in DFSClient delete call
> --
>
> Key: HDFS-13054
> URL: https://issues.apache.org/jira/browse/HDFS-13054
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13054.000.patch, HDFS-13054.001.patch
>
>
> In {{DFSClient#delete}} call, if we get
> {{RemoteException(PathIsNotEmptyDirectoryException)}} we should unwrap and
> throw {{PathIsNotEmptyDirectoryException}} to the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13054) Handling PathIsNotEmptyDirectoryException in DFSClient delete call
[
https://issues.apache.org/jira/browse/HDFS-13054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336331#comment-16336331
]
Nanda kumar commented on HDFS-13054:
Thanks for the review [~arpitagarwal], fixed the same in patch v001.
> Handling PathIsNotEmptyDirectoryException in DFSClient delete call
> --
>
> Key: HDFS-13054
> URL: https://issues.apache.org/jira/browse/HDFS-13054
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13054.000.patch, HDFS-13054.001.patch
>
>
> In {{DFSClient#delete}} call, if we get
> {{RemoteException(PathIsNotEmptyDirectoryException)}} we should unwrap and
> throw {{PathIsNotEmptyDirectoryException}} to the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13054) Handling PathIsNotEmptyDirectoryException in DFSClient delete call
[
https://issues.apache.org/jira/browse/HDFS-13054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar updated HDFS-13054:
---
Attachment: HDFS-13054.001.patch
> Handling PathIsNotEmptyDirectoryException in DFSClient delete call
> --
>
> Key: HDFS-13054
> URL: https://issues.apache.org/jira/browse/HDFS-13054
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13054.000.patch, HDFS-13054.001.patch
>
>
> In {{DFSClient#delete}} call, if we get
> {{RemoteException(PathIsNotEmptyDirectoryException)}} we should unwrap and
> throw {{PathIsNotEmptyDirectoryException}} to the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13017) Block Storage: implement simple iscsi discovery in jscsi server
[ https://issues.apache.org/jira/browse/HDFS-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-13017: Attachment: HDFS-13017-HDFS-7240.004.patch > Block Storage: implement simple iscsi discovery in jscsi server > --- > > Key: HDFS-13017 > URL: https://issues.apache.org/jira/browse/HDFS-13017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HDFS-13017-HDFS-7240.001.patch, > HDFS-13017-HDFS-7240.002.patch, HDFS-13017-HDFS-7240.003.patch, > HDFS-13017-HDFS-7240.004.patch > > > The current jscsi server doesn't support iscsi discovery. > To use jscsi server as a kubernetes storage backend we need the discovery. > jScsi supports it we need just override a method and add an additional call > to the server protocl to get the list of the available cblocks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13054) Handling PathIsNotEmptyDirectoryException in DFSClient delete call
[
https://issues.apache.org/jira/browse/HDFS-13054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336327#comment-16336327
]
Arpit Agarwal commented on HDFS-13054:
--
Thanks for this improvement [~nandakumar131]. Minor comment on the test case:
{code}
fs.create(new Path("/tmp/nonEmptyDir/emptyFile"));
{code}
You can close the FileOutputStream returned here.
+1 otherwise.
> Handling PathIsNotEmptyDirectoryException in DFSClient delete call
> --
>
> Key: HDFS-13054
> URL: https://issues.apache.org/jira/browse/HDFS-13054
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13054.000.patch
>
>
> In {{DFSClient#delete}} call, if we get
> {{RemoteException(PathIsNotEmptyDirectoryException)}} we should unwrap and
> throw {{PathIsNotEmptyDirectoryException}} to the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13054) Handling PathIsNotEmptyDirectoryException in DFSClient delete call
[
https://issues.apache.org/jira/browse/HDFS-13054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336327#comment-16336327
]
Arpit Agarwal edited comment on HDFS-13054 at 1/23/18 8:20 PM:
---
Thanks for this improvement [~nandakumar131]. Minor comment on the test case:
{code}
fs.create(new Path("/tmp/nonEmptyDir/emptyFile"));
{code}
You can immediately close the FileOutputStream returned here. e.g.
_fs.create(new Path("/tmp/nonEmptyDir/emptyFile")).close();_
+1 otherwise.
was (Author: arpitagarwal):
Thanks for this improvement [~nandakumar131]. Minor comment on the test case:
{code}
fs.create(new Path("/tmp/nonEmptyDir/emptyFile"));
{code}
You can close the FileOutputStream returned here.
+1 otherwise.
> Handling PathIsNotEmptyDirectoryException in DFSClient delete call
> --
>
> Key: HDFS-13054
> URL: https://issues.apache.org/jira/browse/HDFS-13054
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13054.000.patch
>
>
> In {{DFSClient#delete}} call, if we get
> {{RemoteException(PathIsNotEmptyDirectoryException)}} we should unwrap and
> throw {{PathIsNotEmptyDirectoryException}} to the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336321#comment-16336321
]
Rakesh R commented on HDFS-13033:
-
Uploaded another patch fixing checkstyle and findbug warnings.
> [SPS]: Implement a mechanism to do file block movements for external SPS
>
>
> Key: HDFS-13033
> URL: https://issues.apache.org/jira/browse/HDFS-13033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Attachments: HDFS-13033-HDFS-10285-00.patch,
> HDFS-13033-HDFS-10285-01.patch, HDFS-13033-HDFS-10285-02.patch,
> HDFS-13033-HDFS-10285-03.patch, HDFS-13033-HDFS-10285-04.patch
>
>
> HDFS-12911 modularization is introducing \{{BlockMoveTaskHandler}} interface
> for moving the file blocks. That will help us to plugin different ways of
> block move mechanisms, if needed.
> For Internal SPS, we have simple blk movement tasks to target DN descriptors.
> For external SPS, we should have mechanism to send \{{replaceBlock}} on
> target node and have a listener to track the block movement completion.
> This is the task to implement the \{{ExternalSPSBlockMoveTaskHandler}} plugin
> for external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-13033:
Attachment: (was: HDFS-13033-HDFS-10285-04.patch)
> [SPS]: Implement a mechanism to do file block movements for external SPS
>
>
> Key: HDFS-13033
> URL: https://issues.apache.org/jira/browse/HDFS-13033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Attachments: HDFS-13033-HDFS-10285-00.patch,
> HDFS-13033-HDFS-10285-01.patch, HDFS-13033-HDFS-10285-02.patch,
> HDFS-13033-HDFS-10285-03.patch, HDFS-13033-HDFS-10285-04.patch
>
>
> HDFS-12911 modularization is introducing \{{BlockMoveTaskHandler}} interface
> for moving the file blocks. That will help us to plugin different ways of
> block move mechanisms, if needed.
> For Internal SPS, we have simple blk movement tasks to target DN descriptors.
> For external SPS, we should have mechanism to send \{{replaceBlock}} on
> target node and have a listener to track the block movement completion.
> This is the task to implement the \{{ExternalSPSBlockMoveTaskHandler}} plugin
> for external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-13033:
Attachment: HDFS-13033-HDFS-10285-04.patch
> [SPS]: Implement a mechanism to do file block movements for external SPS
>
>
> Key: HDFS-13033
> URL: https://issues.apache.org/jira/browse/HDFS-13033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Attachments: HDFS-13033-HDFS-10285-00.patch,
> HDFS-13033-HDFS-10285-01.patch, HDFS-13033-HDFS-10285-02.patch,
> HDFS-13033-HDFS-10285-03.patch, HDFS-13033-HDFS-10285-04.patch
>
>
> HDFS-12911 modularization is introducing \{{BlockMoveTaskHandler}} interface
> for moving the file blocks. That will help us to plugin different ways of
> block move mechanisms, if needed.
> For Internal SPS, we have simple blk movement tasks to target DN descriptors.
> For external SPS, we should have mechanism to send \{{replaceBlock}} on
> target node and have a listener to track the block movement completion.
> This is the task to implement the \{{ExternalSPSBlockMoveTaskHandler}} plugin
> for external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13054) Handling PathIsNotEmptyDirectoryException in DFSClient delete call
[
https://issues.apache.org/jira/browse/HDFS-13054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar updated HDFS-13054:
---
Status: Patch Available (was: Open)
> Handling PathIsNotEmptyDirectoryException in DFSClient delete call
> --
>
> Key: HDFS-13054
> URL: https://issues.apache.org/jira/browse/HDFS-13054
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13054.000.patch
>
>
> In {{DFSClient#delete}} call, if we get
> {{RemoteException(PathIsNotEmptyDirectoryException)}} we should unwrap and
> throw {{PathIsNotEmptyDirectoryException}} to the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13054) Handling PathIsNotEmptyDirectoryException in DFSClient delete call
[
https://issues.apache.org/jira/browse/HDFS-13054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar updated HDFS-13054:
---
Attachment: HDFS-13054.000.patch
> Handling PathIsNotEmptyDirectoryException in DFSClient delete call
> --
>
> Key: HDFS-13054
> URL: https://issues.apache.org/jira/browse/HDFS-13054
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13054.000.patch
>
>
> In {{DFSClient#delete}} call, if we get
> {{RemoteException(PathIsNotEmptyDirectoryException)}} we should unwrap and
> throw {{PathIsNotEmptyDirectoryException}} to the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13033) [SPS]: Implement a mechanism to do file block movements for external SPS
[
https://issues.apache.org/jira/browse/HDFS-13033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-13033:
Attachment: HDFS-13033-HDFS-10285-04.patch
> [SPS]: Implement a mechanism to do file block movements for external SPS
>
>
> Key: HDFS-13033
> URL: https://issues.apache.org/jira/browse/HDFS-13033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Attachments: HDFS-13033-HDFS-10285-00.patch,
> HDFS-13033-HDFS-10285-01.patch, HDFS-13033-HDFS-10285-02.patch,
> HDFS-13033-HDFS-10285-03.patch, HDFS-13033-HDFS-10285-04.patch
>
>
> HDFS-12911 modularization is introducing \{{BlockMoveTaskHandler}} interface
> for moving the file blocks. That will help us to plugin different ways of
> block move mechanisms, if needed.
> For Internal SPS, we have simple blk movement tasks to target DN descriptors.
> For external SPS, we should have mechanism to send \{{replaceBlock}} on
> target node and have a listener to track the block movement completion.
> This is the task to implement the \{{ExternalSPSBlockMoveTaskHandler}} plugin
> for external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12772) RBF: Federation Router State State Store internal API
[
https://issues.apache.org/jira/browse/HDFS-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336303#comment-16336303
]
genericqa commented on HDFS-12772:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 51s{color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs generated 19 new + 375 unchanged
- 19 fixed = 394 total (was 394) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 53s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}177m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestErasureCodingPolicies |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestDFSStripedOutputStream |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestDFSAddressConfig |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStorageStateRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestHDFSFileSystemContract |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12772 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907309/HDFS-12772.005.patch |
| Optional Tests | asflicense compile javac javadoc
[jira] [Commented] (HDFS-13053) Track time to process packet in Datanode
[ https://issues.apache.org/jira/browse/HDFS-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336299#comment-16336299 ] Hanisha Koneru commented on HDFS-13053: --- Hi [~elgoiri] and [~pulkitmisra], Similar work was done in HDFS-10917 and -HDFS-11194-. Peer latency stats are collected during Datanode write pipeline and exposed via metrics. And Namenode detects slow DNs based on these peer stats. > Track time to process packet in Datanode > > > Key: HDFS-13053 > URL: https://issues.apache.org/jira/browse/HDFS-13053 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Íñigo Goiri >Assignee: Pulkit Misra >Priority: Minor > Attachments: HDFS-13053.000.patch > > > We should track the time that each datanode takes to process a packet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13054) Handling PathIsNotEmptyDirectoryException in DFSClient delete call
Nanda kumar created HDFS-13054:
--
Summary: Handling PathIsNotEmptyDirectoryException in DFSClient
delete call
Key: HDFS-13054
URL: https://issues.apache.org/jira/browse/HDFS-13054
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Reporter: Nanda kumar
Assignee: Nanda kumar
In {{DFSClient#delete}} call, if we get
{{RemoteException(PathIsNotEmptyDirectoryException)}} we should unwrap and
throw {{PathIsNotEmptyDirectoryException}} to the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13053) Track time to process packet in Datanode
[ https://issues.apache.org/jira/browse/HDFS-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri reassigned HDFS-13053: -- Assignee: Pulkit Misra > Track time to process packet in Datanode > > > Key: HDFS-13053 > URL: https://issues.apache.org/jira/browse/HDFS-13053 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Íñigo Goiri >Assignee: Pulkit Misra >Priority: Minor > Attachments: HDFS-13053.000.patch > > > We should track the time that each datanode takes to process a packet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13053) Track time to process packet in Datanode
[ https://issues.apache.org/jira/browse/HDFS-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13053: --- Status: Patch Available (was: Open) > Track time to process packet in Datanode > > > Key: HDFS-13053 > URL: https://issues.apache.org/jira/browse/HDFS-13053 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Íñigo Goiri >Assignee: Pulkit Misra >Priority: Minor > Attachments: HDFS-13053.000.patch > > > We should track the time that each datanode takes to process a packet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13053) Track time to process packet in Datanode
[ https://issues.apache.org/jira/browse/HDFS-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336279#comment-16336279 ] Pulkit Misra commented on HDFS-13053: - Added a patch. > Track time to process packet in Datanode > > > Key: HDFS-13053 > URL: https://issues.apache.org/jira/browse/HDFS-13053 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Íñigo Goiri >Priority: Minor > Attachments: HDFS-13053.000.patch > > > We should track the time that each datanode takes to process a packet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13053) Track time to process packet in Datanode
[ https://issues.apache.org/jira/browse/HDFS-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pulkit Misra updated HDFS-13053: Attachment: HDFS-13053.000.patch > Track time to process packet in Datanode > > > Key: HDFS-13053 > URL: https://issues.apache.org/jira/browse/HDFS-13053 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Íñigo Goiri >Priority: Minor > Attachments: HDFS-13053.000.patch > > > We should track the time that each datanode takes to process a packet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12950) [oiv] ls will fail in secure cluster
[
https://issues.apache.org/jira/browse/HDFS-12950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336276#comment-16336276
]
Wei-Chiu Chuang commented on HDFS-12950:
Trying to chime in here.
I feel like implementing a dummy delegation token is the right approach. I
guess this issue can be work-arounded by updating client's authentication to
SIMPLE from KERBEROS.
> [oiv] ls will fail in secure cluster
> -
>
> Key: HDFS-12950
> URL: https://issues.apache.org/jira/browse/HDFS-12950
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
>
> if we execute ls, it will throw following.
> {noformat}
> hdfs dfs -ls webhdfs://127.0.0.1:5978/
> ls: Invalid value for webhdfs parameter "op"
> {noformat}
> When client is configured with security (i.e "hadoop.security.authentication=
> KERBEROS) ,
> then webhdfs will request getdelegation token which is not implemented and
> hence it will throw “ls: Invalid value for webhdfs parameter "op"”.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12997) Move logging to slf4j in BlockPoolSliceStorage and Storage
[ https://issues.apache.org/jira/browse/HDFS-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336267#comment-16336267 ] Ajay Kumar commented on HDFS-12997: --- patch v3 to address checkstyle issues. > Move logging to slf4j in BlockPoolSliceStorage and Storage > --- > > Key: HDFS-12997 > URL: https://issues.apache.org/jira/browse/HDFS-12997 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-12997.001.patch, HDFS-12997.002.patch, > HDFS-12997.003.patch > > > Move logging to slf4j in BlockPoolSliceStorage and Storage classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12997) Move logging to slf4j in BlockPoolSliceStorage and Storage
[ https://issues.apache.org/jira/browse/HDFS-12997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12997: -- Attachment: HDFS-12997.003.patch > Move logging to slf4j in BlockPoolSliceStorage and Storage > --- > > Key: HDFS-12997 > URL: https://issues.apache.org/jira/browse/HDFS-12997 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-12997.001.patch, HDFS-12997.002.patch, > HDFS-12997.003.patch > > > Move logging to slf4j in BlockPoolSliceStorage and Storage classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13053) Track time to process packet in Datanode
[ https://issues.apache.org/jira/browse/HDFS-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336256#comment-16336256 ] Pulkit Misra commented on HDFS-13053: - Yes, I can take this. I have a patch as well, but I am unable to attach it. I don't have the proper permissions, I guess. > Track time to process packet in Datanode > > > Key: HDFS-13053 > URL: https://issues.apache.org/jira/browse/HDFS-13053 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Íñigo Goiri >Priority: Minor > > We should track the time that each datanode takes to process a packet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13053) Track time to process packet in Datanode
[ https://issues.apache.org/jira/browse/HDFS-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336229#comment-16336229 ] Íñigo Goiri commented on HDFS-13053: [~pulkitmisra], you had something already. Do you mind taking this? > Track time to process packet in Datanode > > > Key: HDFS-13053 > URL: https://issues.apache.org/jira/browse/HDFS-13053 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Íñigo Goiri >Priority: Minor > > We should track the time that each datanode takes to process a packet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-13053) Track time to process packet in Datanode
[ https://issues.apache.org/jira/browse/HDFS-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pulkit Misra updated HDFS-13053: Comment: was deleted (was: A patch for the issue.) > Track time to process packet in Datanode > > > Key: HDFS-13053 > URL: https://issues.apache.org/jira/browse/HDFS-13053 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Íñigo Goiri >Priority: Minor > > We should track the time that each datanode takes to process a packet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
