[jira] [Assigned] (HDFS-10854) Remove createStripedFile and addBlockToFile by creating real EC files
[
https://issues.apache.org/jira/browse/HDFS-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
SammiChen reassigned HDFS-10854:
Assignee: SammiChen
> Remove createStripedFile and addBlockToFile by creating real EC files
> -
>
> Key: HDFS-10854
> URL: https://issues.apache.org/jira/browse/HDFS-10854
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, test
>Affects Versions: 3.0.0-alpha2
>Reporter: Zhe Zhang
>Assignee: SammiChen
>
> {{DFSTestUtil#createStripedFile}} and {{addBlockToFile}} were developed
> before we completed EC client. They were used to test the {{NameNode}} EC
> logic when the client was unable to really create/read/write EC files.
> They are causing confusions in other issues about {{NameNode}}. For example,
> in one of the patches under {{HDFS-10301}},
> {{testProcessOverReplicatedAndMissingStripedBlock}} fails because in the test
> we fake a block report from a DN, with a randomly generated storage ID. The
> DN itself is never aware of that storage. This is not possible in a real
> production environment.
> {code}
> DatanodeStorage storage = new
> DatanodeStorage(UUID.randomUUID().toString());
> StorageReceivedDeletedBlocks[] reports = DFSTestUtil
> .makeReportForReceivedBlock(block,
> ReceivedDeletedBlockInfo.BlockStatus.RECEIVING_BLOCK, storage);
> for (StorageReceivedDeletedBlocks report : reports) {
> ns.processIncrementalBlockReport(dn.getDatanodeId(), report);
> }
> {code}
> Now that we have a fully functional EC client, we should remove the old
> testing logic and use similar logic as non-EC tests (creating real files and
> emulate blocks missing / being corrupt).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9668) Optimize the locking in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HDFS-9668:
---
Attachment: HDFS-9668-20.patch
Rebase the code and upload a new patch V20.
The code had been changed since HADOOP-13702 was committed and the rebase,
[~eddyxu] would you mind taking a look at the latest patch again? Thanks a lot!
> Optimize the locking in FsDatasetImpl
> -
>
> Key: HDFS-9668
> URL: https://issues.apache.org/jira/browse/HDFS-9668
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-9668-1.patch, HDFS-9668-10.patch,
> HDFS-9668-11.patch, HDFS-9668-12.patch, HDFS-9668-13.patch,
> HDFS-9668-14.patch, HDFS-9668-14.patch, HDFS-9668-15.patch,
> HDFS-9668-16.patch, HDFS-9668-17.patch, HDFS-9668-18.patch,
> HDFS-9668-19.patch, HDFS-9668-19.patch, HDFS-9668-2.patch,
> HDFS-9668-20.patch, HDFS-9668-3.patch, HDFS-9668-4.patch, HDFS-9668-5.patch,
> HDFS-9668-6.patch, HDFS-9668-7.patch, HDFS-9668-8.patch, HDFS-9668-9.patch,
> execution_time.png
>
>
> During the HBase test on a tiered storage of HDFS (WAL is stored in
> SSD/RAMDISK, and all other files are stored in HDD), we observe many
> long-time BLOCKED threads on FsDatasetImpl in DataNode. The following is part
> of the jstack result:
> {noformat}
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48521 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779272_40852]" - Thread
> t@93336
>java.lang.Thread.State: BLOCKED
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:)
> - waiting to lock <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl) owned by
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" t@93335
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
>
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" - Thread
> t@93335
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:271)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1140)
> - locked <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
> {noformat}
> We measured the execution of some operations in FsDatasetImpl during the
> test. Here following is the result.
> !execution_time.png!
> The operations of finalizeBlock, addBlock and createRbw on HDD in a heavy
> load take a really long time.
> It means one slow operation of finalizeBlock, addBlock and createRbw in a
> slow storage can block all the other same operations in the same DataNode,
> especially in HBase when many wal/flusher/compactor are configured.
>
[jira] [Updated] (HDFS-10684) WebHDFS DataNode calls fail without parameter createparent
[
https://issues.apache.org/jira/browse/HDFS-10684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10684:
--
Status: In Progress (was: Patch Available)
> WebHDFS DataNode calls fail without parameter createparent
> --
>
> Key: HDFS-10684
> URL: https://issues.apache.org/jira/browse/HDFS-10684
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.1
>Reporter: Samuel Low
>Assignee: John Zhuge
>Priority: Blocker
> Labels: compatibility, webhdfs
> Attachments: HDFS-10684.001-branch-2.patch
>
>
> Optional boolean parameters that are not provided in the URL cause the
> WebHDFS create file command to fail.
> curl -i -X PUT
> "http://hadoop-primarynamenode:50070/webhdfs/v1/tmp/test1234?op=CREATE=false;
> Response:
> HTTP/1.1 307 TEMPORARY_REDIRECT
> Cache-Control: no-cache
> Expires: Fri, 15 Jul 2016 04:10:13 GMT
> Date: Fri, 15 Jul 2016 04:10:13 GMT
> Pragma: no-cache
> Expires: Fri, 15 Jul 2016 04:10:13 GMT
> Date: Fri, 15 Jul 2016 04:10:13 GMT
> Pragma: no-cache
> Content-Type: application/octet-stream
> Location:
> http://hadoop-datanode1:50075/webhdfs/v1/tmp/test1234?op=CREATE=hadoop-primarynamenode:8020=false
> Content-Length: 0
> Server: Jetty(6.1.26)
> Following the redirect:
> curl -i -X PUT -T MYFILE
> "http://hadoop-datanode1:50075/webhdfs/v1/tmp/test1234?op=CREATE=hadoop-primarynamenode:8020=false;
> Response:
> HTTP/1.1 100 Continue
> HTTP/1.1 400 Bad Request
> Content-Type: application/json; charset=utf-8
> Content-Length: 162
> Connection: close
>
> {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Failed
> to parse \"null\" to Boolean."}}
> The problem can be circumvented by providing both "createparent" and
> "overwrite" parameters.
> However, this is not possible when I have no control over the WebHDFS calls,
> e.g. Ambari and Hue have errors due to this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10996) Ability to specify per-file EC policy at create time
[ https://issues.apache.org/jira/browse/HDFS-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15621030#comment-15621030 ] SammiChen commented on HDFS-10996: -- Thanks Kai to provide the good idea, I create JIRA HDFS-11075 to track it. > Ability to specify per-file EC policy at create time > > > Key: HDFS-10996 > URL: https://issues.apache.org/jira/browse/HDFS-10996 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen > > Based on discussion in HDFS-10971, it would be useful to specify the EC > policy when the file is created. This is useful for situations where app > requirements do not map nicely to the current directory-level policies. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11075) Provide a tool to help covert existing file of default 3x replication to EC striped layout
SammiChen created HDFS-11075: Summary: Provide a tool to help covert existing file of default 3x replication to EC striped layout Key: HDFS-11075 URL: https://issues.apache.org/jira/browse/HDFS-11075 Project: Hadoop HDFS Issue Type: Task Reporter: SammiChen Assignee: SammiChen Set erasure coding policy to an existing 3x replication file, or change the current existing erasure coding policy of an existing file to another erasure coding policy will force the file to been transformed from one redundancy layout into another one. The converting usually will be time consuming. This task is to provide a new tool or improve an existing tool to facilitate the HDFS administrator to smooth this converting process. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11061) Update dfs -count -t command line help and documentation
[ https://issues.apache.org/jira/browse/HDFS-11061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11061: --- Fix Version/s: (was: 3.0.0-beta1) 3.0.0-alpha2 > Update dfs -count -t command line help and documentation > > > Key: HDFS-11061 > URL: https://issues.apache.org/jira/browse/HDFS-11061 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, fs >Reporter: Wei-Chiu Chuang >Assignee: Yiqun Lin >Priority: Minor > Labels: docuentation, supportability > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-11061.001.patch, HDFS-11061.002.patch, > HDFS-11061.003.patch, HDFS-11061.004.patch > > > According to dfs -count command line help, -t option must be used along with > -q. > * However, the current behavior is that -t can be used without -q, it's just > silently ignored. > * In addition, -t may also be used with -u. > * The FileSystemShell doc does not state -t must be used along with -q. This > should either be enforced in the code, or update the doc/command line. > * Also, the list of possible parameters for -t option is not described in the > doc. Looking at the code (Count.java), the list of possible parameters are > either empty string (="all"), "all", "ram_disk", "ssd", "disk" or "archive" > (caseless) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11072) Add ability to unset and change directory EC policy
[
https://issues.apache.org/jira/browse/HDFS-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620743#comment-15620743
]
Andrew Wang commented on HDFS-11072:
IIUC EC policy is implemented by traversing the target path backwards until we
find a directory with a policy set. This is how storage policies work at least.
Reusing Sammi's example:
{noformat}
mkdir -p /A/B/C
erasurecode -setPolicy thepolicy /A
{noformat}
Files created under A, B, or C all will inherit the policy set on A. If we
wanted to change the policy, we just call {{-setPolicy}} again on A and it will
be picked up by new files. Basically, the default policy is to inherit from the
parent.
bq. In that case, do we need to care about the replication factor. IIUC, the
directory doesn't have the replication factor. Am I missing anything?
You're correct, there isn't a replication factor specified. I just wanted to
clarify with Sammi earlier, since there was a mention of "3x replicated"
specifically, when really it can be general.
bq. Probably we could make this function call simple and apply the changes only
to the given directory. If user wants, then let them iterate and call this
function multiple times.
+1. It might also be nice to provide a CLI tool that prints the policies on the
parent directories of a path.
bq, Instead of adding new "replicated" EC policy, how about call unset EC
policy on a directory and the new files under this will now start writing in
replicated fashion.
Hopefully the above example clarified this, but basically if we want a subdir
of a directory with an EC policy (e.g. {{/A/B/R}}) to create replicated files,
we need to set a policy that's not the default "inherit from parent" policy.
> Add ability to unset and change directory EC policy
> ---
>
> Key: HDFS-11072
> URL: https://issues.apache.org/jira/browse/HDFS-11072
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: SammiChen
> Labels: hdfs-ec-3.0-must-do
>
> Since the directory-level EC policy simply applies to files at create time,
> it makes sense to make it more similar to storage policies and allow changing
> and unsetting the policy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10455) Logging the username when deny the setOwner operation
[
https://issues.apache.org/jira/browse/HDFS-10455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620437#comment-15620437
]
Hudson commented on HDFS-10455:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10729 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10729/])
HDFS-10455. Logging the username when deny the setOwner operation. (brahma: rev
e9c7a97089dd1234a7f1782f79b1b0b190b98f39)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/security/TestPermission.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
> Logging the username when deny the setOwner operation
> -
>
> Key: HDFS-10455
> URL: https://issues.apache.org/jira/browse/HDFS-10455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Rakesh R
>Priority: Minor
> Attachments: HDFS-10455-003.patch, HDFS-10455.000.patch,
> HDFS-10455.002.patch
>
>
> The attached patch appends the user name in the logging when the setOwner
> operation is denied due to insufficient permissions on this user (based on
> his/her name).
> The same practice is used in {{FSPermissionChecker}} such as {{checkOwner()}}
> and {{checkSuperuserPrivilege()}}.
> {code:title=FSDirAttrOp.java|borderStyle=solid}
>if (!pc.isSuperUser()) {
> if (username != null && !pc.getUser().equals(username)) {
> - throw new AccessControlException("Non-super user cannot change
> owner");
> + throw new AccessControlException("User " + pc.getUser()
> + + " is not a super user (non-super user cannot change
> owner).");
> }
> if (group != null && !pc.containsGroup(group)) {
> - throw new AccessControlException("User does not belong to " +
> group);
> + throw new AccessControlException("User " + pc.getUser()
> + + " does not belong to " + group);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10885) [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on
[
https://issues.apache.org/jira/browse/HDFS-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620418#comment-15620418
]
Hadoop QA commented on HDFS-10885:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 9 new + 747 unchanged - 0 fixed = 756 total (was 747) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens |
| | hadoop.hdfs.TestDFSPermission |
| | hadoop.hdfs.server.namenode.TestCheckpoint |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.server.namenode.TestNestedEncryptionZones |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl |
| | hadoop.hdfs.TestDFSUpgrade |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| | hadoop.hdfs.server.namenode.ha.TestHAMetrics |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion |
| | hadoop.hdfs.server.namenode.TestSaveNamespace |
| | hadoop.hdfs.server.datanode.TestDataNodeTcpNoDelay |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer |
| | hadoop.hdfs.TestDataTransferKeepalive |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.namenode.TestINodeFile |
| | hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs |
| | hadoop.hdfs.TestReservedRawPaths |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.namenode.TestAuditLogger |
| | hadoop.hdfs.TestDatanodeLayoutUpgrade |
| | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
| | hadoop.hdfs.server.namenode.TestMetaSave |
| | hadoop.hdfs.TestFileCreation |
| |
[jira] [Commented] (HDFS-10455) Logging the username when deny the setOwner operation
[
https://issues.apache.org/jira/browse/HDFS-10455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620409#comment-15620409
]
Brahma Reddy Battula commented on HDFS-10455:
-
[~rakeshr] thanks for updating the patch.. Committed to trunk.
{{testNonSuperCannotChangeToOtherGroup}} is not there in branch-2, can you
please update branch-2 patch..?
> Logging the username when deny the setOwner operation
> -
>
> Key: HDFS-10455
> URL: https://issues.apache.org/jira/browse/HDFS-10455
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Tianyin Xu
>Assignee: Rakesh R
>Priority: Minor
> Attachments: HDFS-10455-003.patch, HDFS-10455.000.patch,
> HDFS-10455.002.patch
>
>
> The attached patch appends the user name in the logging when the setOwner
> operation is denied due to insufficient permissions on this user (based on
> his/her name).
> The same practice is used in {{FSPermissionChecker}} such as {{checkOwner()}}
> and {{checkSuperuserPrivilege()}}.
> {code:title=FSDirAttrOp.java|borderStyle=solid}
>if (!pc.isSuperUser()) {
> if (username != null && !pc.getUser().equals(username)) {
> - throw new AccessControlException("Non-super user cannot change
> owner");
> + throw new AccessControlException("User " + pc.getUser()
> + + " is not a super user (non-super user cannot change
> owner).");
> }
> if (group != null && !pc.containsGroup(group)) {
> - throw new AccessControlException("User does not belong to " +
> group);
> + throw new AccessControlException("User " + pc.getUser()
> + + " does not belong to " + group);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11031) Add additional unit test for DataNode startup behavior when volumes fail
[
https://issues.apache.org/jira/browse/HDFS-11031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620394#comment-15620394
]
Brahma Reddy Battula commented on HDFS-11031:
-
[~jnp] can you pitch in here(As Ming pinged you first)..? do I go head for
commit..?
> Add additional unit test for DataNode startup behavior when volumes fail
>
>
> Key: HDFS-11031
> URL: https://issues.apache.org/jira/browse/HDFS-11031
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-11031-branch-2.001.patch,
> HDFS-11031-branch-2.002.patch, HDFS-11031-branch-2.003.patch,
> HDFS-11031-branch-2.004.patch, HDFS-11031.000.patch, HDFS-11031.001.patch,
> HDFS-11031.002.patch, HDFS-11031.003.patch, HDFS-11031.004.patch
>
>
> There are several cases to add in {{TestDataNodeVolumeFailure}}:
> - DataNode should not start in case of volumes failure
> - DataNode should not start in case of lacking data dir read/write permission
> - ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11073) FileContext.makeQualified add default port for relavtive path with default fs as ha format
[
https://issues.apache.org/jira/browse/HDFS-11073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620375#comment-15620375
]
Brahma Reddy Battula commented on HDFS-11073:
-
Dupe of HADOOP-12053..?
> FileContext.makeQualified add default port for relavtive path with default
> fs as ha format
> ---
>
> Key: HDFS-11073
> URL: https://issues.apache.org/jira/browse/HDFS-11073
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: DENG FEI
>
> JobHistoryUtils#getPreviousJobHistoryPath used FileContext#makeQualifie
> but history staging dir normally is relatived, the path will add default port
> if "fs.defaultFS" is HA format.It's conflict with FileSystem#checkPath.
> {code}
> Configuration aConf = new Configuration();
> aConf.set("fs.defaultFS", "hdfs://mycluster/");
> System.out.println(FileContext.getFileContext(aConf).makeQualified(new
> Path("/test")));
> {code}
> print hdfs://mycluster:8020/test
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10885) [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on
[ https://issues.apache.org/jira/browse/HDFS-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zhou updated HDFS-10885: Status: Patch Available (was: Open) > [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier > is on > -- > > Key: HDFS-10885 > URL: https://issues.apache.org/jira/browse/HDFS-10885 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Wei Zhou >Assignee: Wei Zhou > Fix For: HDFS-10285 > > Attachments: HDFS-10800-HDFS-10885-00.patch, > HDFS-10800-HDFS-10885-01.patch, HDFS-10800-HDFS-10885-02.patch, > HDFS-10885-HDFS-10285.03.patch, HDFS-10885-HDFS-10285.04.patch, > HDFS-10885-HDFS-10285.05.patch > > > These two can not work at the same time to avoid conflicts and fight with > each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10885) [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on
[
https://issues.apache.org/jira/browse/HDFS-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zhou updated HDFS-10885:
Attachment: HDFS-10885-HDFS-10285.05.patch
As disscussed above, SPS adds an xAttr {{user.creator}} to id file
{{/system/mover.id}} when starting. Thanks!
> [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier
> is on
> --
>
> Key: HDFS-10885
> URL: https://issues.apache.org/jira/browse/HDFS-10885
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-10285
>Reporter: Wei Zhou
>Assignee: Wei Zhou
> Fix For: HDFS-10285
>
> Attachments: HDFS-10800-HDFS-10885-00.patch,
> HDFS-10800-HDFS-10885-01.patch, HDFS-10800-HDFS-10885-02.patch,
> HDFS-10885-HDFS-10285.03.patch, HDFS-10885-HDFS-10285.04.patch,
> HDFS-10885-HDFS-10285.05.patch
>
>
> These two can not work at the same time to avoid conflicts and fight with
> each other.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
