[jira] [Commented] (HDFS-13774) EC: "hdfs ec -getPolicy" is not retrieving policy details when the special REPLICATION policy set on the directory
[ https://issues.apache.org/jira/browse/HDFS-13774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597069#comment-16597069 ] Xiao Chen commented on HDFS-13774: -- bq. For other built-in EC policies, they are disabled by default. This isn't from your change, but we should fix. Out of all the EC policies, RS(6,3) is enabled by default. +1 pending, thanks for the work. > EC: "hdfs ec -getPolicy" is not retrieving policy details when the special > REPLICATION policy set on the directory > -- > > Key: HDFS-13774 > URL: https://issues.apache.org/jira/browse/HDFS-13774 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 > Environment: 3 Node Linux Cluster >Reporter: Souryakanta Dwivedy >Assignee: Ayush Saxena >Priority: Minor > Attachments: GetPolicy_EC.png, HDFS-13774-01.patch > > > Erasure coding: "hdfs ec -getPolicy"" is not retrieving policy details when > the special REPLICATION policy set on the directory > Steps :- > - Create a directory "testEC" > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > - Enable any Erasure coding policy like "XOR-2-1-1024k" > - Set the EC Policy on the Directory > - Get the EC policy for the directory [Received message as : "XOR-2-1-1024k" ] > - Now again set the EC Policy on the directory as "replicate" special > REPLICATION policy > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > The policy is being set for the Directory ,but while retrieving policy > details its throwing error as > policy for the directory is unspecified which is wrong behavior -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-351) Add chill mode state to SCM
[ https://issues.apache.org/jira/browse/HDDS-351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-351: Attachment: HDDS-351.03.patch > Add chill mode state to SCM > --- > > Key: HDDS-351 > URL: https://issues.apache.org/jira/browse/HDDS-351 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-351.00.patch, HDDS-351.01.patch, HDDS-351.02.patch, > HDDS-351.03.patch > > > Add chill mode state to SCM -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-351) Add chill mode state to SCM
[ https://issues.apache.org/jira/browse/HDDS-351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-351: Attachment: (was: HDDS-351.03.patch) > Add chill mode state to SCM > --- > > Key: HDDS-351 > URL: https://issues.apache.org/jira/browse/HDDS-351 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-351.00.patch, HDDS-351.01.patch, HDDS-351.02.patch, > HDDS-351.03.patch > > > Add chill mode state to SCM -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13857) RBF: Choose to enable the default nameservice to write files.
[ https://issues.apache.org/jira/browse/HDFS-13857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597038#comment-16597038 ] yanghuafeng commented on HDFS-13857: Thanks for your comments, I have improved the unit test. Please review again. [~elgoiri] > RBF: Choose to enable the default nameservice to write files. > - > > Key: HDFS-13857 > URL: https://issues.apache.org/jira/browse/HDFS-13857 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation, hdfs >Affects Versions: 3.0.0, 3.1.0, 2.9.1 >Reporter: yanghuafeng >Assignee: yanghuafeng >Priority: Major > Attachments: HDFS-13857.001.patch, HDFS-13857.002.patch, > HDFS-13857.003.patch > > > The default nameservice can provide some default properties for the namenode > protocol. And if we cannot find the path, we will get a location in default > nameservice. From my side as cluster administrator, we need all files to be > written in the location from the MountTableEntry. If no responding location, > some error will return. It is not better to happen some files are written in > some unknown location. We should provide a specific parameter to enable the > default nameservice to store files. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13857) RBF: Choose to enable the default nameservice to write files.
[ https://issues.apache.org/jira/browse/HDFS-13857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanghuafeng updated HDFS-13857: --- Attachment: HDFS-13857.003.patch > RBF: Choose to enable the default nameservice to write files. > - > > Key: HDFS-13857 > URL: https://issues.apache.org/jira/browse/HDFS-13857 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation, hdfs >Affects Versions: 3.0.0, 3.1.0, 2.9.1 >Reporter: yanghuafeng >Assignee: yanghuafeng >Priority: Major > Attachments: HDFS-13857.001.patch, HDFS-13857.002.patch, > HDFS-13857.003.patch > > > The default nameservice can provide some default properties for the namenode > protocol. And if we cannot find the path, we will get a location in default > nameservice. From my side as cluster administrator, we need all files to be > written in the location from the MountTableEntry. If no responding location, > some error will return. It is not better to happen some files are written in > some unknown location. We should provide a specific parameter to enable the > default nameservice to store files. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-372) There are two buffer copies in ChunkOutputStream
[ https://issues.apache.org/jira/browse/HDDS-372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-372: --- Fix Version/s: 0.2.1 > There are two buffer copies in ChunkOutputStream > > > Key: HDDS-372 > URL: https://issues.apache.org/jira/browse/HDDS-372 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-372.20180829.patch > > > Currently, there are two buffer copies in ChunkOutputStream > # from byte[] to ByteBuffer, and > # from ByteBuffer to ByteString. > We should eliminate the ByteBuffer in the middle. > For zero copy io, we should support WritableByteChannel instead of > OutputStream. It won't be done in this JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13852) RBF: The DN_REPORT_TIME_OUT and DN_REPORT_CACHE_EXPIRE should be configured in RBFConfigKeys.
[ https://issues.apache.org/jira/browse/HDFS-13852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597021#comment-16597021 ] yanghuafeng commented on HDFS-13852: Could you review this patch again? [~elgoiri] > RBF: The DN_REPORT_TIME_OUT and DN_REPORT_CACHE_EXPIRE should be configured > in RBFConfigKeys. > - > > Key: HDFS-13852 > URL: https://issues.apache.org/jira/browse/HDFS-13852 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation, hdfs >Affects Versions: 3.1.0, 2.9.1, 3.0.1 >Reporter: yanghuafeng >Assignee: yanghuafeng >Priority: Major > Attachments: HDFS-13852.001.patch, HDFS-13852.002.patch > > > In the NamenodeBeanMetrics the router will invokes 'getDataNodeReport' > periodically. And we can set the dfs.federation.router.dn-report.time-out and > dfs.federation.router.dn-report.cache-expire to avoid time out. But when we > start the router, the FederationMetrics will also invoke the method to get > node usage. If time out error happened, we cannot adjust the parameter > time_out. And the time_out in the FederationMetrics and NamenodeBeanMetrics > should be the same. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13845) RBF: The default MountTableResolver should fail resolving multi-destination paths
[
https://issues.apache.org/jira/browse/HDFS-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597018#comment-16597018
]
yanghuafeng commented on HDFS-13845:
improve some unit tests.
{quote}
The only part that is a little confusing ins getClass() ==
MountTableResolver.class.
What happens when we invoke this from one of the ordered resolvers?
{quote}
getClass() will return the real class of the instance. For example, when you
invoke this method buildLocation from the MultipleDestinationMountableResolver,
it does not equal the MountableResolver.class because you have got the
MultipleDestinationMountableResolver.class
> RBF: The default MountTableResolver should fail resolving multi-destination
> paths
> -
>
> Key: HDFS-13845
> URL: https://issues.apache.org/jira/browse/HDFS-13845
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation, hdfs
>Affects Versions: 3.0.0, 3.1.0, 2.9.1
>Reporter: yanghuafeng
>Assignee: yanghuafeng
>Priority: Major
> Attachments: HDFS-13845.001.patch, HDFS-13845.002.patch
>
>
> When we use the default MountTableResolver to resolve the path, we cannot get
> the destination paths for the default DestinationOrder.HASH.
> {code:java}
> // Some comments here
> private static PathLocation buildLocation(
> ..
> List locations = new LinkedList<>();
> for (RemoteLocation oneDst : entry.getDestinations()) {
> String nsId = oneDst.getNameserviceId();
> String dest = oneDst.getDest();
> String newPath = dest;
> if (!newPath.endsWith(Path.SEPARATOR) && !remainingPath.isEmpty()) {
> newPath += Path.SEPARATOR;
> }
> newPath += remainingPath;
> RemoteLocation remoteLocation = new RemoteLocation(nsId, newPath, path);
> locations.add(remoteLocation);
> }
> DestinationOrder order = entry.getDestOrder();
> return new PathLocation(srcPath, locations, order);
> }
> {code}
> The default order will be hash, but the HashFirstResolver will not be invoked
> to order the location.
> It is ambiguous for the MountTableResolver that we will see the HASH order in
> the web ui for multi-destinations path but we cannot get the result.
> In my opinion, the MountTableResolver will be a simple resolver to implement
> 1 to 1 not including the 1 to n destinations. So we should check the
> buildLocation. If the entry has multi destinations, we should reject it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13863) FsDatasetImpl should log DiskOutOfSpaceException
[
https://issues.apache.org/jira/browse/HDFS-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-13863:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.2
3.0.4
3.2.0
Status: Resolved (was: Patch Available)
Committed this to trunk, branch-3.1 and branch-3.0.
Thanks [~ferhui] for the contribution and thanks [~arpitagarwal], [~jojochuang]
for the reviews.
> FsDatasetImpl should log DiskOutOfSpaceException
>
>
> Key: HDFS-13863
> URL: https://issues.apache.org/jira/browse/HDFS-13863
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.0, 2.9.1, 3.0.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HDFS-13863.001.patch, HDFS-13863.002.patch,
> HDFS-13863.003.patch
>
>
> The code in function *createRbw* as follow
> {code:java}
> try {
> // First try to place the block on a transient volume.
> ref = volumes.getNextTransientVolume(b.getNumBytes());
> datanode.getMetrics().incrRamDiskBlocksWrite();
> } catch (DiskOutOfSpaceException de) {
> // Ignore the exception since we just fall back to persistent
> storage.
> } finally {
> if (ref == null) {
> cacheManager.release(b.getNumBytes());
> }
> }
> {code}
> I think we should log the exception because it took me long time to resolve
> problems, and maybe others face the same problems.
> When i test ram_disk, i found no data was written into randomdisk. I debug,
> deep into the source code, and found that randomdisk size was less than
> reserved space. I think if message was logged, i would resolve the problem
> quickly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13845) RBF: The default MountTableResolver should fail resolving multi-destination paths
[
https://issues.apache.org/jira/browse/HDFS-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yanghuafeng updated HDFS-13845:
---
Attachment: HDFS-13845.002.patch
> RBF: The default MountTableResolver should fail resolving multi-destination
> paths
> -
>
> Key: HDFS-13845
> URL: https://issues.apache.org/jira/browse/HDFS-13845
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation, hdfs
>Affects Versions: 3.0.0, 3.1.0, 2.9.1
>Reporter: yanghuafeng
>Assignee: yanghuafeng
>Priority: Major
> Attachments: HDFS-13845.001.patch, HDFS-13845.002.patch
>
>
> When we use the default MountTableResolver to resolve the path, we cannot get
> the destination paths for the default DestinationOrder.HASH.
> {code:java}
> // Some comments here
> private static PathLocation buildLocation(
> ..
> List locations = new LinkedList<>();
> for (RemoteLocation oneDst : entry.getDestinations()) {
> String nsId = oneDst.getNameserviceId();
> String dest = oneDst.getDest();
> String newPath = dest;
> if (!newPath.endsWith(Path.SEPARATOR) && !remainingPath.isEmpty()) {
> newPath += Path.SEPARATOR;
> }
> newPath += remainingPath;
> RemoteLocation remoteLocation = new RemoteLocation(nsId, newPath, path);
> locations.add(remoteLocation);
> }
> DestinationOrder order = entry.getDestOrder();
> return new PathLocation(srcPath, locations, order);
> }
> {code}
> The default order will be hash, but the HashFirstResolver will not be invoked
> to order the location.
> It is ambiguous for the MountTableResolver that we will see the HASH order in
> the web ui for multi-destinations path but we cannot get the result.
> In my opinion, the MountTableResolver will be a simple resolver to implement
> 1 to 1 not including the 1 to n destinations. So we should check the
> buildLocation. If the entry has multi destinations, we should reject it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-372) There are two buffer copies in ChunkOutputStream
[ https://issues.apache.org/jira/browse/HDDS-372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDDS-372: - Status: Patch Available (was: Open) > There are two buffer copies in ChunkOutputStream > > > Key: HDDS-372 > URL: https://issues.apache.org/jira/browse/HDDS-372 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: HDDS-372.20180829.patch > > > Currently, there are two buffer copies in ChunkOutputStream > # from byte[] to ByteBuffer, and > # from ByteBuffer to ByteString. > We should eliminate the ByteBuffer in the middle. > For zero copy io, we should support WritableByteChannel instead of > OutputStream. It won't be done in this JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-263) Add retries in Ozone Client to handle BLOCK_NOT_COMMITTED Exception
[ https://issues.apache.org/jira/browse/HDDS-263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596940#comment-16596940 ] Tsz Wo Nicholas Sze commented on HDDS-263: -- We should reuse org.apache.hadoop.io.retry.RetryPolicies here. > Add retries in Ozone Client to handle BLOCK_NOT_COMMITTED Exception > --- > > Key: HDDS-263 > URL: https://issues.apache.org/jira/browse/HDDS-263 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Blocker > Fix For: 0.2.1 > > Attachments: HDDS-263.00.patch, HDDS-263.01.patch, HDDS-263.02.patch, > HDDS-263.03.patch > > > While Ozone client writes are going on, a container on a datanode can gets > closed because of node failures, disk out of space etc. In situations as > such, client write will fail with CLOSED_CONTAINER_IO. In this case, ozone > client should try to get the committed block length for the pending open > blocks and update the OzoneManager. While trying to get the committed block > length, it may fail with BLOCK_NOT_COMMITTED exception because as a part of > transiton from CLOSING to CLOSED state for the container , it commits all > open blocks one by one. In such cases, client needs to retry to get the > committed block length for a fixed no of attempts and eventually throw the > exception to the application if its not able to successfully get and update > the length in the OzoneManager. This Jira aims to address this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-372) There are two buffer copies in ChunkOutputStream
[ https://issues.apache.org/jira/browse/HDDS-372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDDS-372: - Attachment: HDDS-372.20180829.patch > There are two buffer copies in ChunkOutputStream > > > Key: HDDS-372 > URL: https://issues.apache.org/jira/browse/HDDS-372 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: HDDS-372.20180829.patch > > > Currently, there are two buffer copies in ChunkOutputStream > # from byte[] to ByteBuffer, and > # from ByteBuffer to ByteString. > We should eliminate the ByteBuffer in the middle. > For zero copy io, we should support WritableByteChannel instead of > OutputStream. It won't be done in this JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-372) There are two buffer copies in ChunkOutputStream
[ https://issues.apache.org/jira/browse/HDDS-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596937#comment-16596937 ] Tsz Wo Nicholas Sze commented on HDDS-372: -- HDDS-372.20180829.patch: 1st patch > There are two buffer copies in ChunkOutputStream > > > Key: HDDS-372 > URL: https://issues.apache.org/jira/browse/HDDS-372 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: HDDS-372.20180829.patch > > > Currently, there are two buffer copies in ChunkOutputStream > # from byte[] to ByteBuffer, and > # from ByteBuffer to ByteString. > We should eliminate the ByteBuffer in the middle. > For zero copy io, we should support WritableByteChannel instead of > OutputStream. It won't be done in this JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13713) Add specification of Multipart Upload API to FS specification, with contract tests
[
https://issues.apache.org/jira/browse/HDFS-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596919#comment-16596919
]
Ewan Higgs commented on HDFS-13713:
---
001
- Basic documentation file in
{{hadoop-common-project/hadoop-common/src/site/markdown/filesystem}}.
- This patch needs help with the specification notation.
> Add specification of Multipart Upload API to FS specification, with contract
> tests
> --
>
> Key: HDFS-13713
> URL: https://issues.apache.org/jira/browse/HDFS-13713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Ewan Higgs
>Priority: Blocker
> Attachments: HDFS-13713.001.patch
>
>
> There's nothing in the FS spec covering the new API. Add it in a new .md file
> * add FS model with the notion of a function mapping (uploadID -> Upload),
> the operations (list, commit, abort). The [TLA+
> mode|https://issues.apache.org/jira/secure/attachment/12865161/objectstore.pdf]l
> of HADOOP-13786 shows how to do this.
> * Contract tests of not just the successful path, but all the invalid ones.
> * implementations of the contract tests of all FSs which support the new API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-380) Remove synchronization from ChunkGroupOutputStream and ChunkOutputStream
[ https://issues.apache.org/jira/browse/HDDS-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596918#comment-16596918 ] Tsz Wo Nicholas Sze commented on HDDS-380: -- Not surprisingly, all tests pass without any further changes after removed "synchronized" from ChunkOutputStream and ChunkGroupOutputStream. Thanks, [~shashikant]! > Remove synchronization from ChunkGroupOutputStream and ChunkOutputStream > > > Key: HDDS-380 > URL: https://issues.apache.org/jira/browse/HDDS-380 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-380.00.patch > > > In one of the code review for HDDS-247, it was suggested that > ChunkGroupOutputStream/ChunkOutputStream may not be thread safe as the Java > OutputStream subclasses are generally non-thread-safe for performance > reasons. If users want thread safe, they can easily synchronize it > themselves. This Jira aims to remove synchronization from > ChunkGroupOutputStream and ChunkOutputStream. > cc [~szetszwo]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13713) Add specification of Multipart Upload API to FS specification, with contract tests
[ https://issues.apache.org/jira/browse/HDFS-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HDFS-13713: -- Attachment: HDFS-13713.001.patch > Add specification of Multipart Upload API to FS specification, with contract > tests > -- > > Key: HDFS-13713 > URL: https://issues.apache.org/jira/browse/HDFS-13713 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, test >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ewan Higgs >Priority: Blocker > Attachments: HDFS-13713.001.patch > > > There's nothing in the FS spec covering the new API. Add it in a new .md file > * add FS model with the notion of a function mapping (uploadID -> Upload), > the operations (list, commit, abort). The [TLA+ > mode|https://issues.apache.org/jira/secure/attachment/12865161/objectstore.pdf]l > of HADOOP-13786 shows how to do this. > * Contract tests of not just the successful path, but all the invalid ones. > * implementations of the contract tests of all FSs which support the new API. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-108) Update Node2ContainerMap while processing container reports
[ https://issues.apache.org/jira/browse/HDDS-108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-108. --- Resolution: Implemented > Update Node2ContainerMap while processing container reports > --- > > Key: HDDS-108 > URL: https://issues.apache.org/jira/browse/HDDS-108 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-108.00.patch, HDDS-108.01.patch, HDDS-108.02.patch, > HDDS-108.03.patch, HDDS-108.04.patch > > > When the container report comes, the Node2Container Map should be updated via > SCMContainerManager. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-132) Update SCMNodeStorageStatMap while processing Node Report
[ https://issues.apache.org/jira/browse/HDDS-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDDS-132. --- Resolution: Implemented > Update SCMNodeStorageStatMap while processing Node Report > - > > Key: HDDS-132 > URL: https://issues.apache.org/jira/browse/HDDS-132 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Blocker > Fix For: 0.3.0 > > > When the node report is received at SCM, SCMNodeStorageStatMap needs to get > updated. > In the event of a node/Volume failure, this Map needs to be updated as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-337) keys created with key name having special character/wildcard should not allowed
[
https://issues.apache.org/jira/browse/HDDS-337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596897#comment-16596897
]
Dinesh Chitlangia edited comment on HDDS-337 at 8/29/18 10:23 PM:
--
[~nilotpalnandi] - Have you started working on this Jira ? If not, I would like
to take this as I have a working version.
was (Author: dineshchitlangia):
[~nilotpalnandi] - Have you started working on this Jira ? If not, I would like
to take this.
> keys created with key name having special character/wildcard should not
> allowed
> ---
>
> Key: HDDS-337
> URL: https://issues.apache.org/jira/browse/HDDS-337
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Reporter: Nilotpal Nandi
>Assignee: Nilotpal Nandi
>Priority: Major
> Fix For: 0.2.1
>
>
> Please find the snippet of command execution. Here , the keys are created
> with wildcard special character in its key name.
> Expectation :
> wildcard special characters should not be allowed.
>
> {noformat}
> hadoop@1a1fa8a11332:~/bin$ ./ozone oz -putKey root-volume/root-bucket/d++
> -file /etc/services -v
> 2018-08-08 13:17:48 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Volume Name : root-volume
> Bucket Name : root-bucket
> Key Name : d++
> File Hash : 567c100888518c1163b3462993de7d47
> Key Name : d++ does not exist, creating it
> 2018-08-08 13:17:48 INFO ConfUtils:41 - raft.rpc.type = GRPC (default)
> 2018-08-08 13:17:48 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432
> (custom)
> 2018-08-08 13:17:48 INFO ConfUtils:41 - raft.client.rpc.retryInterval = 300
> ms (default)
> 2018-08-08 13:17:48 INFO ConfUtils:41 -
> raft.client.async.outstanding-requests.max = 100 (default)
> 2018-08-08 13:17:48 INFO ConfUtils:41 - raft.client.async.scheduler-threads =
> 3 (default)
> 2018-08-08 13:17:49 INFO ConfUtils:41 - raft.grpc.flow.control.window = 1MB
> (=1048576) (default)
> 2018-08-08 13:17:49 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432
> (custom)
> 2018-08-08 13:17:49 INFO ConfUtils:41 - raft.client.rpc.request.timeout =
> 3000 ms (default)
> Aug 08, 2018 1:17:49 PM
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl detectProxy
> WARNING: Failed to construct URI for proxy lookup, proceeding without proxy
> java.net.URISyntaxException: Illegal character in hostname at index 13:
> https://ozone_datanode_1.ozone_default:9858
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.parseHostname(URI.java:3387)
> at java.net.URI$Parser.parseServer(URI.java:3236)
> at java.net.URI$Parser.parseAuthority(URI.java:3155)
> at java.net.URI$Parser.parseHierarchical(URI.java:3097)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:673)
> at
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.detectProxy(ProxyDetectorImpl.java:128)
> at
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.proxyFor(ProxyDetectorImpl.java:118)
> at
> org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:207)
> at
> org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.obtainActiveTransport(InternalSubchannel.java:188)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$SubchannelImpl.requestConnection(ManagedChannelImpl.java:1130)
> at
> org.apache.ratis.shaded.io.grpc.PickFirstBalancerFactory$PickFirstBalancer.handleResolvedAddressGroups(PickFirstBalancerFactory.java:79)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl$1NamesResolved.run(ManagedChannelImpl.java:1032)
> at
> org.apache.ratis.shaded.io.grpc.internal.ChannelExecutor.drain(ChannelExecutor.java:73)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$LbHelperImpl.runSerialized(ManagedChannelImpl.java:1000)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl.onAddresses(ManagedChannelImpl.java:1044)
> at
> org.apache.ratis.shaded.io.grpc.internal.DnsNameResolver$1.run(DnsNameResolver.java:201)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> hadoop@1a1fa8a11332:~/bin$ ./ozone oz -putKey root-volume/root-bucket/d**
> -file /etc/passwd -v
> 2018-08-08 13:18:13 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Volume Name : root-volume
> Bucket Name : root-bucket
> Key Name : d**
> File Hash :
[jira] [Commented] (HDDS-337) keys created with key name having special character/wildcard should not allowed
[
https://issues.apache.org/jira/browse/HDDS-337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596897#comment-16596897
]
Dinesh Chitlangia commented on HDDS-337:
[~nilotpalnandi] - Have you started working on this Jira ? If not, I would like
to take this.
> keys created with key name having special character/wildcard should not
> allowed
> ---
>
> Key: HDDS-337
> URL: https://issues.apache.org/jira/browse/HDDS-337
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Reporter: Nilotpal Nandi
>Assignee: Nilotpal Nandi
>Priority: Major
> Fix For: 0.2.1
>
>
> Please find the snippet of command execution. Here , the keys are created
> with wildcard special character in its key name.
> Expectation :
> wildcard special characters should not be allowed.
>
> {noformat}
> hadoop@1a1fa8a11332:~/bin$ ./ozone oz -putKey root-volume/root-bucket/d++
> -file /etc/services -v
> 2018-08-08 13:17:48 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Volume Name : root-volume
> Bucket Name : root-bucket
> Key Name : d++
> File Hash : 567c100888518c1163b3462993de7d47
> Key Name : d++ does not exist, creating it
> 2018-08-08 13:17:48 INFO ConfUtils:41 - raft.rpc.type = GRPC (default)
> 2018-08-08 13:17:48 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432
> (custom)
> 2018-08-08 13:17:48 INFO ConfUtils:41 - raft.client.rpc.retryInterval = 300
> ms (default)
> 2018-08-08 13:17:48 INFO ConfUtils:41 -
> raft.client.async.outstanding-requests.max = 100 (default)
> 2018-08-08 13:17:48 INFO ConfUtils:41 - raft.client.async.scheduler-threads =
> 3 (default)
> 2018-08-08 13:17:49 INFO ConfUtils:41 - raft.grpc.flow.control.window = 1MB
> (=1048576) (default)
> 2018-08-08 13:17:49 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432
> (custom)
> 2018-08-08 13:17:49 INFO ConfUtils:41 - raft.client.rpc.request.timeout =
> 3000 ms (default)
> Aug 08, 2018 1:17:49 PM
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl detectProxy
> WARNING: Failed to construct URI for proxy lookup, proceeding without proxy
> java.net.URISyntaxException: Illegal character in hostname at index 13:
> https://ozone_datanode_1.ozone_default:9858
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.parseHostname(URI.java:3387)
> at java.net.URI$Parser.parseServer(URI.java:3236)
> at java.net.URI$Parser.parseAuthority(URI.java:3155)
> at java.net.URI$Parser.parseHierarchical(URI.java:3097)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:673)
> at
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.detectProxy(ProxyDetectorImpl.java:128)
> at
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.proxyFor(ProxyDetectorImpl.java:118)
> at
> org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:207)
> at
> org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.obtainActiveTransport(InternalSubchannel.java:188)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$SubchannelImpl.requestConnection(ManagedChannelImpl.java:1130)
> at
> org.apache.ratis.shaded.io.grpc.PickFirstBalancerFactory$PickFirstBalancer.handleResolvedAddressGroups(PickFirstBalancerFactory.java:79)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl$1NamesResolved.run(ManagedChannelImpl.java:1032)
> at
> org.apache.ratis.shaded.io.grpc.internal.ChannelExecutor.drain(ChannelExecutor.java:73)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$LbHelperImpl.runSerialized(ManagedChannelImpl.java:1000)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl.onAddresses(ManagedChannelImpl.java:1044)
> at
> org.apache.ratis.shaded.io.grpc.internal.DnsNameResolver$1.run(DnsNameResolver.java:201)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> hadoop@1a1fa8a11332:~/bin$ ./ozone oz -putKey root-volume/root-bucket/d**
> -file /etc/passwd -v
> 2018-08-08 13:18:13 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Volume Name : root-volume
> Bucket Name : root-bucket
> Key Name : d**
> File Hash : b056233571cc80d6879212911cb8e500
> Key Name : d** does not exist, creating it
> 2018-08-08 13:18:14 INFO ConfUtils:41 - raft.rpc.type = GRPC (default)
> 2018-08-08 13:18:14 INFO ConfUtils:41 -
[jira] [Commented] (HDFS-10755) TestDecommissioningStatus BindException Failure
[
https://issues.apache.org/jira/browse/HDFS-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596881#comment-16596881
]
Eric Badger commented on HDFS-10755:
[~kennychang] were you actually able to reproduce the error when the patch is
applied? This patch is from a few years ago so I don't remember the analysis.
But it looks like it goes out and set the port in the conf to grab an ephemeral
port. So I'm not sure why that would fail with a port bind issue.
> TestDecommissioningStatus BindException Failure
> ---
>
> Key: HDFS-10755
> URL: https://issues.apache.org/jira/browse/HDFS-10755
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: HDFS-10755.001.patch, HDFS-10755.002.patch
>
>
> Tests in TestDecomissioningStatus call MiniDFSCluster.dataNodeRestart(). They
> are required to come back up on the same (initially ephemeral) port that they
> were on before being shutdown. Because of this, there is an inherent race
> condition where another process could bind to the port while the datanode is
> down. If this happens then we get a BindException failure. However, all of
> the tests in TestDecommissioningStatus depend on the cluster being up and
> running for them to run correctly. So if a test blows up the cluster, the
> subsequent tests will also fail. Below I show the BindException failure as
> well as the subsequent test failure that occurred.
> {noformat}
> java.net.BindException: Problem binding to [localhost:35370]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:436)
> at sun.nio.ch.Net.bind(Net.java:428)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:430)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:768)
> at org.apache.hadoop.ipc.Server.(Server.java:2391)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:951)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:523)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:498)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:796)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initIpcServer(DataNode.java:802)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1134)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:429)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2387)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2274)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2321)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartDataNode(MiniDFSCluster.java:2037)
> at
> org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus.testDecommissionDeadDN(TestDecommissioningStatus.java:426)
> {noformat}
> {noformat}
> java.lang.AssertionError: Number of Datanodes expected:<2> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus.testDecommissionStatus(TestDecommissioningStatus.java:275)
> {noformat}
> I don't think there's any way to avoid the inherent race condition with
> getting the same ephemeral port, but we can definitely fix the tests so that
> it doesn't cause subsequent tests to fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13879) FileSystem: Should allowSnapshot() and disallowSnapshot() be part of it?
[ https://issues.apache.org/jira/browse/HDFS-13879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596880#comment-16596880 ] Wei-Chiu Chuang commented on HDFS-13879: Thanks [~smeng] for initiating the discussion. Would you please update the affected versions? [~ste...@apache.org] could you help enlighten us here? As I recall, there was some discussion regarding FileSystem APIs but I don't recall the details (nor could I find the jiras) > FileSystem: Should allowSnapshot() and disallowSnapshot() be part of it? > > > Key: HDFS-13879 > URL: https://issues.apache.org/jira/browse/HDFS-13879 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Siyao Meng >Priority: Major > > I wonder whether we should add allowSnapshot() and disallowSnapshot() to > FileSystem abstract class. > I think we should because createSnapshot(), renameSnapshot() and > deleteSnapshot() are already part of it. > Any reason why we don't want to do this? > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10755) TestDecommissioningStatus BindException Failure
[
https://issues.apache.org/jira/browse/HDFS-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596875#comment-16596875
]
Kenny Chang commented on HDFS-10755:
It still applies! This patch reduces the failure rate, but does not eliminate
it. It is simply trying one more time. It is not perfect, still an improvement.
> TestDecommissioningStatus BindException Failure
> ---
>
> Key: HDFS-10755
> URL: https://issues.apache.org/jira/browse/HDFS-10755
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: HDFS-10755.001.patch, HDFS-10755.002.patch
>
>
> Tests in TestDecomissioningStatus call MiniDFSCluster.dataNodeRestart(). They
> are required to come back up on the same (initially ephemeral) port that they
> were on before being shutdown. Because of this, there is an inherent race
> condition where another process could bind to the port while the datanode is
> down. If this happens then we get a BindException failure. However, all of
> the tests in TestDecommissioningStatus depend on the cluster being up and
> running for them to run correctly. So if a test blows up the cluster, the
> subsequent tests will also fail. Below I show the BindException failure as
> well as the subsequent test failure that occurred.
> {noformat}
> java.net.BindException: Problem binding to [localhost:35370]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:436)
> at sun.nio.ch.Net.bind(Net.java:428)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:430)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:768)
> at org.apache.hadoop.ipc.Server.(Server.java:2391)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:951)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:523)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:498)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:796)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initIpcServer(DataNode.java:802)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1134)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:429)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2387)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2274)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2321)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartDataNode(MiniDFSCluster.java:2037)
> at
> org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus.testDecommissionDeadDN(TestDecommissioningStatus.java:426)
> {noformat}
> {noformat}
> java.lang.AssertionError: Number of Datanodes expected:<2> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus.testDecommissionStatus(TestDecommissioningStatus.java:275)
> {noformat}
> I don't think there's any way to avoid the inherent race condition with
> getting the same ephemeral port, but we can definitely fix the tests so that
> it doesn't cause subsequent tests to fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
[ https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596872#comment-16596872 ] Dinesh Chitlangia edited comment on HDDS-98 at 8/29/18 9:30 PM: [~anu], [~xyao] - Request you to please review the patch. cc: [~jnp] - your inputs have been incorporated. was (Author: dineshchitlangia): [~anu], [~xyao] - Request you to please review the patch. > Adding Ozone Manager Audit Log > -- > > Key: HDDS-98 > URL: https://issues.apache.org/jira/browse/HDDS-98 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Dinesh Chitlangia >Priority: Major > Labels: Logging, audit > Fix For: 0.2.1 > > Attachments: HDDS-98.001.patch, HDDS-98.002.patch, HDDS-98.003.patch, > HDDS-98.004.patch, HDDS-98.005.patch, audit.log, log4j2.properties > > > This ticket is opened to add ozone manager's audit log. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-98) Adding Ozone Manager Audit Log
[ https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HDDS-98: -- Attachment: HDDS-98.005.patch Status: Patch Available (was: In Progress) [~anu], [~xyao] - Request you to please review the patch. > Adding Ozone Manager Audit Log > -- > > Key: HDDS-98 > URL: https://issues.apache.org/jira/browse/HDDS-98 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Dinesh Chitlangia >Priority: Major > Labels: Logging, audit > Fix For: 0.2.1 > > Attachments: HDDS-98.001.patch, HDDS-98.002.patch, HDDS-98.003.patch, > HDDS-98.004.patch, HDDS-98.005.patch, audit.log, log4j2.properties > > > This ticket is opened to add ozone manager's audit log. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-370) Add and implement following functions in SCMClientProtocolServer
[ https://issues.apache.org/jira/browse/HDDS-370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-370: Description: Add and implement following functions in SCMClientProtocolServer # isScmInChillMode # forceScmExitChillMode was: Add and implement following functions in SCMClientProtocolServer # isScmInChillMode # forceScmEnterChillMode # forceScmExitChillMode > Add and implement following functions in SCMClientProtocolServer > > > Key: HDDS-370 > URL: https://issues.apache.org/jira/browse/HDDS-370 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > > Add and implement following functions in SCMClientProtocolServer > # isScmInChillMode > # forceScmExitChillMode -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-379) Simplify and improve the cli arg parsing of ozone scmcli
[
https://issues.apache.org/jira/browse/HDDS-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596851#comment-16596851
]
Ajay Kumar commented on HDDS-379:
-
[~elek] thanks for updating the patch. Patch looks good to me. Few comments:
* SCMCli doesn't utilize {{GenericOptionsParser}} which allows users to pass
config value from cli. (i.e -D) , either we can implement Tool interface or
directly use {{GenericOptionsParser}} to enable this.
* Rename
** {{CreateSubcommand}} to {{CreateSubCommand}}
** {{CloseSubcommand}} to {{CloseSubCommand}}
** {{DeleteSubcommand}} to {{DeleteSubCommand}}
** {{InfoSubcommand}} to {{InfoSubCommand}}
** {{ListSubcommand}} to {{ListSubCommand}}
* SCMCLI: Scm Cli can be used for other commands other than container as well.
Ex to exit SCM out of chill mode.
** L73 change {{"ozone scm cli container"}} to {{"ozone scm cli"}}
** L74 change {{Developer tools to handle container specific}} to {{Developer
tools to handle SCM specific}}
* Seems specifying invalid scm host doesn't print any error or help message
even with verbose option. {{bin/ozone scmcli --scm=abc:8020 --verbose}}
However it does show error when we add subcommand (but still not the full stack
trace, error is good enough in this case but in some cases full stack trace
might be handy):
{code}bin/ozone scmcli --scm=abc:8020 --verbose list --start=0
Invalid host name: local host is: (unknown); destination host is: "abc":8020;
java.net.UnknownHostException; For more details see:
http://wiki.apache.org/hadoop/UnknownHost{code}
> Simplify and improve the cli arg parsing of ozone scmcli
>
>
> Key: HDDS-379
> URL: https://issues.apache.org/jira/browse/HDDS-379
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-379.001.patch, HDDS-379.002.patch
>
>
> SCMCLI is a useful tool to test SCM. It can create/delete/close/list
> containers.
> There are multiple problems with the current scmcli.
> The biggest one is the cli argument handling. Similar to HDDS-190, it's often
> very hard to get the help for a specific subcommand.
> The other one is that a big part of the code is the argument handling which
> is mixed with the business logic.
> I propose to use a more modern argument handler library and simplify the
> argument handling (and improve the user experience).
> I propose to use [picocli|https://github.com/remkop/picocli].
> 1.) It supports subcommands and subcommand specific and general arguments.
> 2.) It could work based on annotation with very few additional boilerplate
> code
> 3.) It's very well documented and easy to use
> 4.) It's licenced under Apache licence
> 5.) It supports tab autocompletion for bash and zsh and colorful output
> 6.) Actively maintainer project
> 7.) Adopter by other bigger projects (groovy, junit, log4j)
> In this patch I would like to demonstrate how the cli handling could be
> simplified. And if it's accepted, we can start to use similar approach for
> other ozone cli as well.
> The patch also fixes the cli (the name of the main class was wrong).
> It also requires HDDS-377 for the be compiled.
> I also deleted the TestSCMCli. It was turned off with an annotation and I
> believe that this functionality could be tested more easily with a robot test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-280) Support ozone dist-start-stitching on openbsd/osx
[
https://issues.apache.org/jira/browse/HDDS-280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDDS-280:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the contribution [~elek] and thanks [~anu] and [~aw] for the
reviews. I have committed this to trunk.
> Support ozone dist-start-stitching on openbsd/osx
> -
>
> Key: HDDS-280
> URL: https://issues.apache.org/jira/browse/HDDS-280
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-280.001.patch, HDDS-280.002.patch,
> HDDS-280.003.patch
>
>
> {quote}Ozone is creating a symlink during the dist process.
> Using the "ozone" directory as a destination name all the docker-based
> acceptance tests and docker-compose files are more simple as they don't need
> to have the version information in the path.
> But to keep the version specific folder name in the tar file we create a
> symbolic link during the tar creation. With the symbolic link and the
> '–dereference' tar argument we can create the tar file which includes a
> versioned directory (ozone-0.2.1) but we can use the a dist directory without
> the version in the name (hadoop-dist/target/ozone).
> {quote}
> This is the description of the current
> dev-support/bin/ozone-dist-tar-stitching. [~aw] in a comment for HDDS-276
> pointed to the problem that some bsd variants don't support the dereference
> command line options of the ln command.
> The main reason to use this approach is to get a simplified destination name
> without the version (hadoop-dist/target/ozone instead of
> hadoop-dist/target/ozone-0.2.1). It simplifies the docker-compose based
> environments and acceptance tests, therefore I prefer to keep the simplified
> destination name.
> The issue is the tar file creation, if and only if we need the version number
> in the name of the root directory inside of the tar.
> Possible solutions:
> # Use cp target/ozone target/ozone-0.2.1 + tar. It's simple but more slow
> and requires more space.
> # Do the tar distribution from docker all the time in case of 'dereference'
> is not supported. Not very convenient
> # Accept that tar will contain ozone directory and not ozone-0.2.1. This is
> the more simple and can be improved with an additional VERSION file in the
> root of the distribution.
> # (+1) Use hadoop-dist/target/ozone-0.2.1 instead of
> hadoop-dist/target/ozone. This is more complex for the docker based testing
> as we need the explicit names in the compose files (volume:
> ../../../hadoop-dist/target/ozone-0.2.1). The structure is more complex with
> using version in the directory name.
> Please comment your preference.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13846) Safe blocks counter is not decremented correctly if the block is striped
[
https://issues.apache.org/jira/browse/HDFS-13846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596832#comment-16596832
]
Daniel Templeton commented on HDFS-13846:
-
I see. That makes sense. Might be nice to add a comment to explain that so
that someone doesn't "fix" it later by making the conditional test {{<=}}.
Aside from that, LGTM. Did you look at the deprecation warning that popped up?
The jenkins build is gone now, so I can't verify whether it was related to
code you added.
> Safe blocks counter is not decremented correctly if the block is striped
>
>
> Key: HDFS-13846
> URL: https://issues.apache.org/jira/browse/HDFS-13846
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.0
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Major
> Attachments: HDFS-13846.001.patch, HDFS-13846.002.patch,
> HDFS-13846.003.patch
>
>
> In BlockManagerSafeMode class, the "safe blocks" counter is incremented if
> the number of nodes containing the block equals to the number of data units
> specified by the erasure coding policy, which looks like this in the code:
> {code:java}
> final int safe = storedBlock.isStriped() ?
> ((BlockInfoStriped)storedBlock).getRealDataBlockNum() :
> safeReplication;
> if (storageNum == safe) {
> this.blockSafe++;
> {code}
> But when it is decremented the code does not check if the block is striped or
> not, just compares the number of nodes containing the block with 0
> (safeReplication - 1) if the block is complete, which is not correct.
> {code:java}
> if (storedBlock.isComplete() &&
> blockManager.countNodes(b).liveReplicas() == safeReplication - 1) {
> this.blockSafe--;
> assert blockSafe >= 0;
> checkSafeMode();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13830) Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list
[ https://issues.apache.org/jira/browse/HDFS-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596819#comment-16596819 ] Daniel Templeton commented on HDFS-13830: - Actually, the typo goes the other direction. A compatible change is always allowed in a maintenance release, no matter what the stability label. It's compatible, so it's not an issue. > Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting > snasphottable directory list > > > Key: HDFS-13830 > URL: https://issues.apache.org/jira/browse/HDFS-13830 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.0.3 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13830.branch-3.0.001.patch, > HDFS-13830.branch-3.0.002.patch, HDFS-13830.branch-3.0.003.patch, > HDFS-13830.branch-3.0.004.patch > > > HDFS-13141 conflicts with 3.0.3 because of interface change in HdfsFileStatus. > This Jira aims to backport the WebHDFS getSnapshottableDirListing() support > to branch-3.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13830) Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list
[ https://issues.apache.org/jira/browse/HDFS-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596816#comment-16596816 ] Wei-Chiu Chuang commented on HDFS-13830: +1 Thanks [~smeng]. Also checked compat guide line since this patch changes the visibility of methods and static class fields of FileStatus, which is widely used and Public.Stable, it is essentially a compatible change. Under Hadoop compatibility guideline [http://hadoop.apache.org/docs/r3.0.3/hadoop-project-dist/hadoop-common/InterfaceClassification.html#Stable,] a compat change can be made between maintenance releases. (Note there's a typo being addressed by HADOOP-15705) (Wish Yetus could detect Hadoop compatibility violation) > Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting > snasphottable directory list > > > Key: HDFS-13830 > URL: https://issues.apache.org/jira/browse/HDFS-13830 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.0.3 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13830.branch-3.0.001.patch, > HDFS-13830.branch-3.0.002.patch, HDFS-13830.branch-3.0.003.patch, > HDFS-13830.branch-3.0.004.patch > > > HDFS-13141 conflicts with 3.0.3 because of interface change in HdfsFileStatus. > This Jira aims to backport the WebHDFS getSnapshottableDirListing() support > to branch-3.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-13830) Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list
[ https://issues.apache.org/jira/browse/HDFS-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-13830: --- Comment: was deleted (was: -1 this patch violates Hadoop compat guidelines. Thanks [~smeng] but since this patch changes the visibility of methods and static class fields of FileStatus, which is widely used and Public.Stable, it is essentially a compatible change. Under Hadoop compatibility guideline [http://hadoop.apache.org/docs/r3.0.3/hadoop-project-dist/hadoop-common/InterfaceClassification.html#Stable,] a compat change can only be made between minor releases. To enforce a better discipline I suggest we close this Jira as won't fix. (Wish Yetus can detect Hadoop compatibility violation)) > Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting > snasphottable directory list > > > Key: HDFS-13830 > URL: https://issues.apache.org/jira/browse/HDFS-13830 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.0.3 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13830.branch-3.0.001.patch, > HDFS-13830.branch-3.0.002.patch, HDFS-13830.branch-3.0.003.patch, > HDFS-13830.branch-3.0.004.patch > > > HDFS-13141 conflicts with 3.0.3 because of interface change in HdfsFileStatus. > This Jira aims to backport the WebHDFS getSnapshottableDirListing() support > to branch-3.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13830) Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list
[ https://issues.apache.org/jira/browse/HDFS-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596787#comment-16596787 ] Siyao Meng commented on HDFS-13830: --- [~jojochuang] Yeah. Thanks anyway! > Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting > snasphottable directory list > > > Key: HDFS-13830 > URL: https://issues.apache.org/jira/browse/HDFS-13830 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.0.3 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13830.branch-3.0.001.patch, > HDFS-13830.branch-3.0.002.patch, HDFS-13830.branch-3.0.003.patch, > HDFS-13830.branch-3.0.004.patch > > > HDFS-13141 conflicts with 3.0.3 because of interface change in HdfsFileStatus. > This Jira aims to backport the WebHDFS getSnapshottableDirListing() support > to branch-3.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13830) Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list
[ https://issues.apache.org/jira/browse/HDFS-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-13830: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) > Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting > snasphottable directory list > > > Key: HDFS-13830 > URL: https://issues.apache.org/jira/browse/HDFS-13830 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.0.3 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13830.branch-3.0.001.patch, > HDFS-13830.branch-3.0.002.patch, HDFS-13830.branch-3.0.003.patch, > HDFS-13830.branch-3.0.004.patch > > > HDFS-13141 conflicts with 3.0.3 because of interface change in HdfsFileStatus. > This Jira aims to backport the WebHDFS getSnapshottableDirListing() support > to branch-3.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13830) Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list
[ https://issues.apache.org/jira/browse/HDFS-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596783#comment-16596783 ] Wei-Chiu Chuang commented on HDFS-13830: -1 this patch violates Hadoop compat guidelines. Thanks [~smeng] but since this patch changes the visibility of methods and static class fields of FileStatus, which is widely used and Public.Stable, it is essentially a compatible change. Under Hadoop compatibility guideline [http://hadoop.apache.org/docs/r3.0.3/hadoop-project-dist/hadoop-common/InterfaceClassification.html#Stable,] a compat change can only be made between minor releases. To enforce a better discipline I suggest we close this Jira as won't fix. (Wish Yetus can detect Hadoop compatibility violation) > Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting > snasphottable directory list > > > Key: HDFS-13830 > URL: https://issues.apache.org/jira/browse/HDFS-13830 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.0.3 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13830.branch-3.0.001.patch, > HDFS-13830.branch-3.0.002.patch, HDFS-13830.branch-3.0.003.patch, > HDFS-13830.branch-3.0.004.patch > > > HDFS-13141 conflicts with 3.0.3 because of interface change in HdfsFileStatus. > This Jira aims to backport the WebHDFS getSnapshottableDirListing() support > to branch-3.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-280) Support ozone dist-start-stitching on openbsd/osx
[
https://issues.apache.org/jira/browse/HDDS-280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596761#comment-16596761
]
Mukul Kumar Singh commented on HDDS-280:
Thanks for working on this [~elek]. The patch looks good to me as well.
+1 for the v3 patch, there is a minor nitpick in acceptance-test/pom.xml:47,
the indentation is off by 1.
I will handle this during commit.
> Support ozone dist-start-stitching on openbsd/osx
> -
>
> Key: HDDS-280
> URL: https://issues.apache.org/jira/browse/HDDS-280
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-280.001.patch, HDDS-280.002.patch,
> HDDS-280.003.patch
>
>
> {quote}Ozone is creating a symlink during the dist process.
> Using the "ozone" directory as a destination name all the docker-based
> acceptance tests and docker-compose files are more simple as they don't need
> to have the version information in the path.
> But to keep the version specific folder name in the tar file we create a
> symbolic link during the tar creation. With the symbolic link and the
> '–dereference' tar argument we can create the tar file which includes a
> versioned directory (ozone-0.2.1) but we can use the a dist directory without
> the version in the name (hadoop-dist/target/ozone).
> {quote}
> This is the description of the current
> dev-support/bin/ozone-dist-tar-stitching. [~aw] in a comment for HDDS-276
> pointed to the problem that some bsd variants don't support the dereference
> command line options of the ln command.
> The main reason to use this approach is to get a simplified destination name
> without the version (hadoop-dist/target/ozone instead of
> hadoop-dist/target/ozone-0.2.1). It simplifies the docker-compose based
> environments and acceptance tests, therefore I prefer to keep the simplified
> destination name.
> The issue is the tar file creation, if and only if we need the version number
> in the name of the root directory inside of the tar.
> Possible solutions:
> # Use cp target/ozone target/ozone-0.2.1 + tar. It's simple but more slow
> and requires more space.
> # Do the tar distribution from docker all the time in case of 'dereference'
> is not supported. Not very convenient
> # Accept that tar will contain ozone directory and not ozone-0.2.1. This is
> the more simple and can be improved with an additional VERSION file in the
> root of the distribution.
> # (+1) Use hadoop-dist/target/ozone-0.2.1 instead of
> hadoop-dist/target/ozone. This is more complex for the docker based testing
> as we need the explicit names in the compose files (volume:
> ../../../hadoop-dist/target/ozone-0.2.1). The structure is more complex with
> using version in the directory name.
> Please comment your preference.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13830) Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list
[ https://issues.apache.org/jira/browse/HDFS-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596748#comment-16596748 ] Siyao Meng edited comment on HDFS-13830 at 8/29/18 7:08 PM: [~jojochuang] Thanks for the review! But those two lines are already removed in rev 004. was (Author: smeng): [~jojochuang] Thanks for the review! > Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting > snasphottable directory list > > > Key: HDFS-13830 > URL: https://issues.apache.org/jira/browse/HDFS-13830 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.0.3 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13830.branch-3.0.001.patch, > HDFS-13830.branch-3.0.002.patch, HDFS-13830.branch-3.0.003.patch, > HDFS-13830.branch-3.0.004.patch > > > HDFS-13141 conflicts with 3.0.3 because of interface change in HdfsFileStatus. > This Jira aims to backport the WebHDFS getSnapshottableDirListing() support > to branch-3.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13830) Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list
[ https://issues.apache.org/jira/browse/HDFS-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596748#comment-16596748 ] Siyao Meng edited comment on HDFS-13830 at 8/29/18 7:05 PM: [~jojochuang] Thanks for the review! was (Author: smeng): [~jojochuang] Thanks for the review! But those two lines have already been removed in rev 004. > Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting > snasphottable directory list > > > Key: HDFS-13830 > URL: https://issues.apache.org/jira/browse/HDFS-13830 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.0.3 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13830.branch-3.0.001.patch, > HDFS-13830.branch-3.0.002.patch, HDFS-13830.branch-3.0.003.patch, > HDFS-13830.branch-3.0.004.patch > > > HDFS-13141 conflicts with 3.0.3 because of interface change in HdfsFileStatus. > This Jira aims to backport the WebHDFS getSnapshottableDirListing() support > to branch-3.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13830) Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list
[ https://issues.apache.org/jira/browse/HDFS-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596748#comment-16596748 ] Siyao Meng commented on HDFS-13830: --- [~jojochuang] Thanks for the review! But those two lines have already been removed in rev 004. > Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting > snasphottable directory list > > > Key: HDFS-13830 > URL: https://issues.apache.org/jira/browse/HDFS-13830 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.0.3 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13830.branch-3.0.001.patch, > HDFS-13830.branch-3.0.002.patch, HDFS-13830.branch-3.0.003.patch, > HDFS-13830.branch-3.0.004.patch > > > HDFS-13141 conflicts with 3.0.3 because of interface change in HdfsFileStatus. > This Jira aims to backport the WebHDFS getSnapshottableDirListing() support > to branch-3.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13695) Move logging to slf4j in HDFS package
[ https://issues.apache.org/jira/browse/HDFS-13695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Pickering updated HDFS-13695: - Attachment: HDFS-13695.v10.patch > Move logging to slf4j in HDFS package > - > > Key: HDFS-13695 > URL: https://issues.apache.org/jira/browse/HDFS-13695 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Ian Pickering >Priority: Major > Attachments: HDFS-13695.v1.patch, HDFS-13695.v10.patch, > HDFS-13695.v2.patch, HDFS-13695.v3.patch, HDFS-13695.v4.patch, > HDFS-13695.v5.patch, HDFS-13695.v6.patch, HDFS-13695.v7.patch, > HDFS-13695.v8.patch, HDFS-13695.v9.patch > > > Move logging to slf4j in HDFS package -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13830) Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list
[
https://issues.apache.org/jira/browse/HDFS-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596609#comment-16596609
]
Wei-Chiu Chuang commented on HDFS-13830:
The patch looks mostly good. There are two extra lines in the test though:
{code:title=TestWebHDFS#testWebHdfsSnapshottableDirectoryList}
SnapshottableDirectoryStatus[] statuses =
webHdfs.getSnapshottableDirectoryList();
Assert.assertNull(statuses);{code}
> Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting
> snasphottable directory list
>
>
> Key: HDFS-13830
> URL: https://issues.apache.org/jira/browse/HDFS-13830
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 3.0.3
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-13830.branch-3.0.001.patch,
> HDFS-13830.branch-3.0.002.patch, HDFS-13830.branch-3.0.003.patch,
> HDFS-13830.branch-3.0.004.patch
>
>
> HDFS-13141 conflicts with 3.0.3 because of interface change in HdfsFileStatus.
> This Jira aims to backport the WebHDFS getSnapshottableDirListing() support
> to branch-3.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13816) dfs.getQuotaUsage() throws NPE on non-existent dir instead of FileNotFoundException
[
https://issues.apache.org/jira/browse/HDFS-13816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596596#comment-16596596
]
Vinayakumar B commented on HDFS-13816:
--
Thanks [~shashikant] and [~knanasi].
Updated test. Please review
> dfs.getQuotaUsage() throws NPE on non-existent dir instead of
> FileNotFoundException
> ---
>
> Key: HDFS-13816
> URL: https://issues.apache.org/jira/browse/HDFS-13816
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Major
> Attachments: HDFS-13816-01.patch, HDFS-13816-02.patch
>
>
> {{dfs.getQuotaUsage()}} on non-existent path should throw
> FileNotFoundException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getQuotaUsageInt(FSDirStatAndListingOp.java:573)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getQuotaUsage(FSDirStatAndListingOp.java:554)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getQuotaUsage(FSNamesystem.java:3221)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getQuotaUsage(NameNodeRpcServer.java:1404)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getQuotaUsage(ClientNamenodeProtocolServerSideTranslatorPB.java:1861)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13816) dfs.getQuotaUsage() throws NPE on non-existent dir instead of FileNotFoundException
[
https://issues.apache.org/jira/browse/HDFS-13816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-13816:
-
Attachment: HDFS-13816-02.patch
> dfs.getQuotaUsage() throws NPE on non-existent dir instead of
> FileNotFoundException
> ---
>
> Key: HDFS-13816
> URL: https://issues.apache.org/jira/browse/HDFS-13816
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Major
> Attachments: HDFS-13816-01.patch, HDFS-13816-02.patch
>
>
> {{dfs.getQuotaUsage()}} on non-existent path should throw
> FileNotFoundException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getQuotaUsageInt(FSDirStatAndListingOp.java:573)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getQuotaUsage(FSDirStatAndListingOp.java:554)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getQuotaUsage(FSNamesystem.java:3221)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getQuotaUsage(NameNodeRpcServer.java:1404)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getQuotaUsage(ClientNamenodeProtocolServerSideTranslatorPB.java:1861)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-351) Add chill mode state to SCM
[ https://issues.apache.org/jira/browse/HDDS-351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-351: Status: Open (was: Patch Available) > Add chill mode state to SCM > --- > > Key: HDDS-351 > URL: https://issues.apache.org/jira/browse/HDDS-351 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-351.00.patch, HDDS-351.01.patch, HDDS-351.02.patch, > HDDS-351.03.patch > > > Add chill mode state to SCM -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-351) Add chill mode state to SCM
[ https://issues.apache.org/jira/browse/HDDS-351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-351: Status: Patch Available (was: Open) > Add chill mode state to SCM > --- > > Key: HDDS-351 > URL: https://issues.apache.org/jira/browse/HDDS-351 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-351.00.patch, HDDS-351.01.patch, HDDS-351.02.patch, > HDDS-351.03.patch > > > Add chill mode state to SCM -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13869) RBF: Handle NPE for NamenodeBeanMetrics#getFederationMetrics()
[
https://issues.apache.org/jira/browse/HDFS-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596555#comment-16596555
]
Íñigo Goiri commented on HDFS-13869:
>From the trace, it looks like the null part is {{getFederationMetrics()}}, so
>it's not like router is null but {{this.router.getMetrics()}}.
I'm not completely sure what the issue is here.
We would also need a unit test to reproduce your particular case.
> RBF: Handle NPE for NamenodeBeanMetrics#getFederationMetrics()
> --
>
> Key: HDFS-13869
> URL: https://issues.apache.org/jira/browse/HDFS-13869
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-13869.patch
>
>
> {code:java}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getUsed(NamenodeBeanMetrics.java:205)
> at
> org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getCapacityUsed(NamenodeBeanMetrics.java:519)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
> ngMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13867) RBF: Add validation for max arguments for Router admin ls, clrQuota, setQuota, rm and nameservice commands
[ https://issues.apache.org/jira/browse/HDFS-13867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596553#comment-16596553 ] Íñigo Goiri commented on HDFS-13867: [^HDFS-13867-05.patch] LGTM. +1 pending a clean Yetus report. > RBF: Add validation for max arguments for Router admin ls, clrQuota, > setQuota, rm and nameservice commands > -- > > Key: HDFS-13867 > URL: https://issues.apache.org/jira/browse/HDFS-13867 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-13867-01.patch, HDFS-13867-02.patch, > HDFS-13867-03.patch, HDFS-13867-04.patch, HDFS-13867-05.patch > > > Add validation to check if the total number of arguments provided for the > Router Admin commands are not more than max possible.In most cases if there > are some non related extra parameters after the required arguments it doesn't > validate against this but instead perform the action with the required > parameters and ignore the extra ones which shouldn't be in the ideal case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13867) RBF: Add validation for max arguments for Router admin ls, clrQuota, setQuota, rm and nameservice commands
[
https://issues.apache.org/jira/browse/HDFS-13867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592956#comment-16592956
]
Íñigo Goiri edited comment on HDFS-13867 at 8/29/18 4:10 PM:
-
I was trying to see how {{DFSAdmin}} was handling this and it looks like it
doesn't; I would like to keep it consistent but both seem pretty ad-hoc.
Anyway, minor comment: can we make the new method private and add a small
javadoc too?
was (Author: elgoiri):
I was trying to see how {[DFSAdmin}} was handling this and it looks like it
doesn't; I would like to keep it consistent but both seem pretty ad-hoc.
Anyway, minor comment: can we make the new method private and add a small
javadoc too?
> RBF: Add validation for max arguments for Router admin ls, clrQuota,
> setQuota, rm and nameservice commands
> --
>
> Key: HDFS-13867
> URL: https://issues.apache.org/jira/browse/HDFS-13867
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-13867-01.patch, HDFS-13867-02.patch,
> HDFS-13867-03.patch, HDFS-13867-04.patch, HDFS-13867-05.patch
>
>
> Add validation to check if the total number of arguments provided for the
> Router Admin commands are not more than max possible.In most cases if there
> are some non related extra parameters after the required arguments it doesn't
> validate against this but instead perform the action with the required
> parameters and ignore the extra ones which shouldn't be in the ideal case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13697) DFSClient should instantiate and cache KMSClientProvider using UGI at creation time for consistent UGI handling
[
https://issues.apache.org/jira/browse/HDFS-13697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596512#comment-16596512
]
Zsolt Venczel commented on HDFS-13697:
--
Thanks [~daryn] for the investigation and explanation and thanks [~xiaochen]
for the continuous work and discussion!
In my latest patch (09) I addressed the following:
* *KMSClientProvider*
No more morphing... The doAsUser is calculated at construction time and that's
it.
* *TestKMS*
Based on Xiao's findings I fixed the key provider creation in the
doProxyUserTest function to correctly test key creation by proxy users.
* *TestAclsEndToEnd*
I think the main issue with this test suite was that it was using the mini
cluster dfs client for all of its operations. As we stopped morphing the
problem had surfaced therefore I refactored it to use a truly end-to-end
approach by having a proper client and a proper, client side key provider. The
fat part of the changes are due to introducing a service user that needed the
appropriate ACLs for the various testing scenarios.
> DFSClient should instantiate and cache KMSClientProvider using UGI at
> creation time for consistent UGI handling
> ---
>
> Key: HDFS-13697
> URL: https://issues.apache.org/jira/browse/HDFS-13697
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zsolt Venczel
>Assignee: Zsolt Venczel
>Priority: Major
> Attachments: HDFS-13697.01.patch, HDFS-13697.02.patch,
> HDFS-13697.03.patch, HDFS-13697.04.patch, HDFS-13697.05.patch,
> HDFS-13697.06.patch, HDFS-13697.07.patch, HDFS-13697.08.patch,
> HDFS-13697.09.patch, HDFS-13697.prelim.patch
>
>
> While calling KeyProviderCryptoExtension decryptEncryptedKey the call stack
> might not have doAs privileged execution call (in the DFSClient for example).
> This results in loosing the proxy user from UGI as UGI.getCurrentUser finds
> no AccessControllerContext and does a re-login for the login user only.
> This can cause the following for example: if we have set up the oozie user to
> be entitled to perform actions on behalf of example_user but oozie is
> forbidden to decrypt any EDEK (for security reasons), due to the above issue,
> example_user entitlements are lost from UGI and the following error is
> reported:
> {code}
> [0]
> SERVER[xxx] USER[example_user] GROUP[-] TOKEN[] APP[Test_EAR]
> JOB[0020905-180313191552532-oozie-oozi-W]
> ACTION[0020905-180313191552532-oozie-oozi-W@polling_dir_path] Error starting
> action [polling_dir_path]. ErrorType [ERROR], ErrorCode [FS014], Message
> [FS014: User [oozie] is not authorized to perform [DECRYPT_EEK] on key with
> ACL name [encrypted_key]!!]
> org.apache.oozie.action.ActionExecutorException: FS014: User [oozie] is not
> authorized to perform [DECRYPT_EEK] on key with ACL name [encrypted_key]!!
> at
> org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:463)
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:441)
> at
> org.apache.oozie.action.hadoop.FsActionExecutor.touchz(FsActionExecutor.java:523)
> at
> org.apache.oozie.action.hadoop.FsActionExecutor.doOperations(FsActionExecutor.java:199)
> at
> org.apache.oozie.action.hadoop.FsActionExecutor.start(FsActionExecutor.java:563)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
> at org.apache.oozie.command.XCommand.call(XCommand.java:286)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User
> [oozie] is not authorized to perform [DECRYPT_EEK] on key with ACL name
> [encrypted_key]!!
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
>
[jira] [Updated] (HDFS-13697) DFSClient should instantiate and cache KMSClientProvider using UGI at creation time for consistent UGI handling
[
https://issues.apache.org/jira/browse/HDFS-13697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zsolt Venczel updated HDFS-13697:
-
Attachment: HDFS-13697.09.patch
> DFSClient should instantiate and cache KMSClientProvider using UGI at
> creation time for consistent UGI handling
> ---
>
> Key: HDFS-13697
> URL: https://issues.apache.org/jira/browse/HDFS-13697
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zsolt Venczel
>Assignee: Zsolt Venczel
>Priority: Major
> Attachments: HDFS-13697.01.patch, HDFS-13697.02.patch,
> HDFS-13697.03.patch, HDFS-13697.04.patch, HDFS-13697.05.patch,
> HDFS-13697.06.patch, HDFS-13697.07.patch, HDFS-13697.08.patch,
> HDFS-13697.09.patch, HDFS-13697.prelim.patch
>
>
> While calling KeyProviderCryptoExtension decryptEncryptedKey the call stack
> might not have doAs privileged execution call (in the DFSClient for example).
> This results in loosing the proxy user from UGI as UGI.getCurrentUser finds
> no AccessControllerContext and does a re-login for the login user only.
> This can cause the following for example: if we have set up the oozie user to
> be entitled to perform actions on behalf of example_user but oozie is
> forbidden to decrypt any EDEK (for security reasons), due to the above issue,
> example_user entitlements are lost from UGI and the following error is
> reported:
> {code}
> [0]
> SERVER[xxx] USER[example_user] GROUP[-] TOKEN[] APP[Test_EAR]
> JOB[0020905-180313191552532-oozie-oozi-W]
> ACTION[0020905-180313191552532-oozie-oozi-W@polling_dir_path] Error starting
> action [polling_dir_path]. ErrorType [ERROR], ErrorCode [FS014], Message
> [FS014: User [oozie] is not authorized to perform [DECRYPT_EEK] on key with
> ACL name [encrypted_key]!!]
> org.apache.oozie.action.ActionExecutorException: FS014: User [oozie] is not
> authorized to perform [DECRYPT_EEK] on key with ACL name [encrypted_key]!!
> at
> org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:463)
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:441)
> at
> org.apache.oozie.action.hadoop.FsActionExecutor.touchz(FsActionExecutor.java:523)
> at
> org.apache.oozie.action.hadoop.FsActionExecutor.doOperations(FsActionExecutor.java:199)
> at
> org.apache.oozie.action.hadoop.FsActionExecutor.start(FsActionExecutor.java:563)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
> at org.apache.oozie.command.XCommand.call(XCommand.java:286)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User
> [oozie] is not authorized to perform [DECRYPT_EEK] on key with ACL name
> [encrypted_key]!!
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:607)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:565)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:832)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:209)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:205)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:205)
> at
>
[jira] [Commented] (HDFS-13818) Extend OIV to detect FSImage corruption
[ https://issues.apache.org/jira/browse/HDFS-13818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596456#comment-16596456 ] Adam Antal commented on HDFS-13818: --- Uploaded patch v2. Incorporated the changes [~gabor.bota] you wrote, they were very helpful indeed. Also run additional checkstyle for fixing (new) errors. Added unit tests (set-get and simple checkers) for the created Corruption class and whole functional tests for the new processor. In fact, how the processor works and the usage haven't changed, so I refer the reviewer to the still valid doc found in the uploads. Only minor things have changed (update it later) like extra column was added to the output. > Extend OIV to detect FSImage corruption > --- > > Key: HDFS-13818 > URL: https://issues.apache.org/jira/browse/HDFS-13818 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: HDFS-13818.001.patch, HDFS-13818.002.patch, > OIV_CorruptionDetector_processor.001.pdf > > > A follow-up Jira for HDFS-13031: an improvement of the OIV is suggested for > detecting corruptions like HDFS-13101 in an offline way. > The reasoning is the following. Apart from a NN startup throwing the error, > there is nothing in the customer's hand that could reassure him/her that the > FSImages is good or corrupted. > Although real full checking of the FSImage is only possible by the NN, for > stack traces associated with the observed corruption cases the solution of > putting up a tertiary NN is a little bit of overkill. The OIV would be a > handy choice, already having functionality like loading the fsimage and > constructing the folder structure, we just have to add the option of > detecting the null INodes. For e.g. the Delimited OIV processor can already > use in disk MetadataMap, which reduces memory consumption. Also there may be > a window for parallelizing: iterating through INodes for e.g. could be done > distributed, increasing efficiency, and we wouldn't need a high mem-high CPU > setup for just checking the FSImage. > The suggestion is to add a --detectCorruption option to the OIV which would > check the FSImage for consistency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13818) Extend OIV to detect FSImage corruption
[ https://issues.apache.org/jira/browse/HDFS-13818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated HDFS-13818: -- Attachment: HDFS-13818.002.patch > Extend OIV to detect FSImage corruption > --- > > Key: HDFS-13818 > URL: https://issues.apache.org/jira/browse/HDFS-13818 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: HDFS-13818.001.patch, HDFS-13818.002.patch, > OIV_CorruptionDetector_processor.001.pdf > > > A follow-up Jira for HDFS-13031: an improvement of the OIV is suggested for > detecting corruptions like HDFS-13101 in an offline way. > The reasoning is the following. Apart from a NN startup throwing the error, > there is nothing in the customer's hand that could reassure him/her that the > FSImages is good or corrupted. > Although real full checking of the FSImage is only possible by the NN, for > stack traces associated with the observed corruption cases the solution of > putting up a tertiary NN is a little bit of overkill. The OIV would be a > handy choice, already having functionality like loading the fsimage and > constructing the folder structure, we just have to add the option of > detecting the null INodes. For e.g. the Delimited OIV processor can already > use in disk MetadataMap, which reduces memory consumption. Also there may be > a window for parallelizing: iterating through INodes for e.g. could be done > distributed, increasing efficiency, and we wouldn't need a high mem-high CPU > setup for just checking the FSImage. > The suggestion is to add a --detectCorruption option to the OIV which would > check the FSImage for consistency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13863) FsDatasetImpl should log DiskOutOfSpaceException
[
https://issues.apache.org/jira/browse/HDFS-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596362#comment-16596362
]
Arpit Agarwal commented on HDFS-13863:
--
+1
> FsDatasetImpl should log DiskOutOfSpaceException
>
>
> Key: HDFS-13863
> URL: https://issues.apache.org/jira/browse/HDFS-13863
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.0, 2.9.1, 3.0.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-13863.001.patch, HDFS-13863.002.patch,
> HDFS-13863.003.patch
>
>
> The code in function *createRbw* as follow
> {code:java}
> try {
> // First try to place the block on a transient volume.
> ref = volumes.getNextTransientVolume(b.getNumBytes());
> datanode.getMetrics().incrRamDiskBlocksWrite();
> } catch (DiskOutOfSpaceException de) {
> // Ignore the exception since we just fall back to persistent
> storage.
> } finally {
> if (ref == null) {
> cacheManager.release(b.getNumBytes());
> }
> }
> {code}
> I think we should log the exception because it took me long time to resolve
> problems, and maybe others face the same problems.
> When i test ram_disk, i found no data was written into randomdisk. I debug,
> deep into the source code, and found that randomdisk size was less than
> reserved space. I think if message was logged, i would resolve the problem
> quickly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13867) RBF: Add validation for max arguments for Router admin ls, clrQuota, setQuota, rm and nameservice commands
[ https://issues.apache.org/jira/browse/HDFS-13867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-13867: Attachment: HDFS-13867-05.patch > RBF: Add validation for max arguments for Router admin ls, clrQuota, > setQuota, rm and nameservice commands > -- > > Key: HDFS-13867 > URL: https://issues.apache.org/jira/browse/HDFS-13867 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-13867-01.patch, HDFS-13867-02.patch, > HDFS-13867-03.patch, HDFS-13867-04.patch, HDFS-13867-05.patch > > > Add validation to check if the total number of arguments provided for the > Router Admin commands are not more than max possible.In most cases if there > are some non related extra parameters after the required arguments it doesn't > validate against this but instead perform the action with the required > parameters and ignore the extra ones which shouldn't be in the ideal case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-325) Add event watcher for delete blocks command
[ https://issues.apache.org/jira/browse/HDDS-325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596352#comment-16596352 ] Lokesh Jain commented on HDDS-325: -- Uploaded rebased v6 patch which addresses [~elek] comments. I have modified the test case in TestBlockDeletion to verify the event to be fired by watcher. > Add event watcher for delete blocks command > --- > > Key: HDDS-325 > URL: https://issues.apache.org/jira/browse/HDDS-325 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-325.001.patch, HDDS-325.002.patch, > HDDS-325.003.patch, HDDS-325.004.patch, HDDS-325.005.patch, HDDS-325.006.patch > > > This Jira aims to add watcher for deleteBlocks command. It removes the > current rpc call required for datanode to send the acknowledgement for > deleteBlocks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-325) Add event watcher for delete blocks command
[ https://issues.apache.org/jira/browse/HDDS-325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDDS-325: - Attachment: HDDS-325.006.patch > Add event watcher for delete blocks command > --- > > Key: HDDS-325 > URL: https://issues.apache.org/jira/browse/HDDS-325 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-325.001.patch, HDDS-325.002.patch, > HDDS-325.003.patch, HDDS-325.004.patch, HDDS-325.005.patch, HDDS-325.006.patch > > > This Jira aims to add watcher for deleteBlocks command. It removes the > current rpc call required for datanode to send the acknowledgement for > deleteBlocks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13882) Change dfs.client.block.write.locateFollowingBlock.retries default from 5 to 10
[ https://issues.apache.org/jira/browse/HDFS-13882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kitti Nanasi updated HDFS-13882: Status: Patch Available (was: Open) > Change dfs.client.block.write.locateFollowingBlock.retries default from 5 to > 10 > --- > > Key: HDFS-13882 > URL: https://issues.apache.org/jira/browse/HDFS-13882 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Attachments: HDFS-13882.001.patch > > > More and more we are seeing cases where customers are running into the java > io exception "Unable to close file because the last block does not have > enough number of replicas" on client file closure. The common workaround is > to increase dfs.client.block.write.locateFollowingBlock.retries from 5 to 10. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13882) Change dfs.client.block.write.locateFollowingBlock.retries default from 5 to 10
[ https://issues.apache.org/jira/browse/HDFS-13882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kitti Nanasi updated HDFS-13882: Attachment: HDFS-13882.001.patch > Change dfs.client.block.write.locateFollowingBlock.retries default from 5 to > 10 > --- > > Key: HDFS-13882 > URL: https://issues.apache.org/jira/browse/HDFS-13882 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Attachments: HDFS-13882.001.patch > > > More and more we are seeing cases where customers are running into the java > io exception "Unable to close file because the last block does not have > enough number of replicas" on client file closure. The common workaround is > to increase dfs.client.block.write.locateFollowingBlock.retries from 5 to 10. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-379) Simplify and improve the cli arg parsing of ozone scmcli
[ https://issues.apache.org/jira/browse/HDDS-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596284#comment-16596284 ] Elek, Marton commented on HDDS-379: --- v002 addresses the findbug issue. > Simplify and improve the cli arg parsing of ozone scmcli > > > Key: HDDS-379 > URL: https://issues.apache.org/jira/browse/HDDS-379 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-379.001.patch, HDDS-379.002.patch > > > SCMCLI is a useful tool to test SCM. It can create/delete/close/list > containers. > There are multiple problems with the current scmcli. > The biggest one is the cli argument handling. Similar to HDDS-190, it's often > very hard to get the help for a specific subcommand. > The other one is that a big part of the code is the argument handling which > is mixed with the business logic. > I propose to use a more modern argument handler library and simplify the > argument handling (and improve the user experience). > I propose to use [picocli|https://github.com/remkop/picocli]. > 1.) It supports subcommands and subcommand specific and general arguments. > 2.) It could work based on annotation with very few additional boilerplate > code > 3.) It's very well documented and easy to use > 4.) It's licenced under Apache licence > 5.) It supports tab autocompletion for bash and zsh and colorful output > 6.) Actively maintainer project > 7.) Adopter by other bigger projects (groovy, junit, log4j) > In this patch I would like to demonstrate how the cli handling could be > simplified. And if it's accepted, we can start to use similar approach for > other ozone cli as well. > The patch also fixes the cli (the name of the main class was wrong). > It also requires HDDS-377 for the be compiled. > I also deleted the TestSCMCli. It was turned off with an annotation and I > believe that this functionality could be tested more easily with a robot test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-379) Simplify and improve the cli arg parsing of ozone scmcli
[ https://issues.apache.org/jira/browse/HDDS-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-379: -- Attachment: HDDS-379.002.patch > Simplify and improve the cli arg parsing of ozone scmcli > > > Key: HDDS-379 > URL: https://issues.apache.org/jira/browse/HDDS-379 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-379.001.patch, HDDS-379.002.patch > > > SCMCLI is a useful tool to test SCM. It can create/delete/close/list > containers. > There are multiple problems with the current scmcli. > The biggest one is the cli argument handling. Similar to HDDS-190, it's often > very hard to get the help for a specific subcommand. > The other one is that a big part of the code is the argument handling which > is mixed with the business logic. > I propose to use a more modern argument handler library and simplify the > argument handling (and improve the user experience). > I propose to use [picocli|https://github.com/remkop/picocli]. > 1.) It supports subcommands and subcommand specific and general arguments. > 2.) It could work based on annotation with very few additional boilerplate > code > 3.) It's very well documented and easy to use > 4.) It's licenced under Apache licence > 5.) It supports tab autocompletion for bash and zsh and colorful output > 6.) Actively maintainer project > 7.) Adopter by other bigger projects (groovy, junit, log4j) > In this patch I would like to demonstrate how the cli handling could be > simplified. And if it's accepted, we can start to use similar approach for > other ozone cli as well. > The patch also fixes the cli (the name of the main class was wrong). > It also requires HDDS-377 for the be compiled. > I also deleted the TestSCMCli. It was turned off with an annotation and I > believe that this functionality could be tested more easily with a robot test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-343) Containers are stuck in closing state in scm
[
https://issues.apache.org/jira/browse/HDDS-343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596283#comment-16596283
]
Nanda kumar commented on HDDS-343:
--
As part of HDDS-364, the variable {{datanodeState}} is changed to {{contInfo}}
in {{ContainerMapping#processContainerReports}}. This is causing compilation
failure. [~elek], can you please fix this?
> Containers are stuck in closing state in scm
>
>
> Key: HDDS-343
> URL: https://issues.apache.org/jira/browse/HDDS-343
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Fix For: 0.2.1
>
> Attachments: HDDS-343.001.patch, HDDS-343.002.patch,
> HDDS-343.003.patch, HDDS-343.004.patch, HDDS-343.005.patch
>
>
> Containers could not been closed currently.
> The datanode is closing the containers and sending the CLOSED state in the
> container report but SCM doesn't register that the state is closed and
> sending the close command again and again.
> I think the ContainerMapping.processContainerReport should be improved.
> {code}
> scm_1 | --> RPC message request: SCMHeartbeatRequestProto from
> 172.25.0.2:33912
> scm_1 | datanodeDetails {
> scm_1 | uuid: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 | ipAddress: "172.25.0.2"
> scm_1 | hostName: "365fd1f44f0b"
> scm_1 | ports {
> scm_1 | name: "STANDALONE"
> scm_1 | value: 9859
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "RATIS"
> scm_1 | value: 9858
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "REST"
> scm_1 | value: 9880
> scm_1 | }
> scm_1 | }
> scm_1 | nodeReport {
> scm_1 | storageReport {
> scm_1 | storageUuid: "DS-61e76107-85c5-437a-95a7-aeb8b3e7827f"
> scm_1 | storageLocation: "/tmp/hadoop-hadoop/dfs/data"
> scm_1 | capacity: 491630870528
> scm_1 | scmUsed: 2708828160
> scm_1 | remaining: 24263614464
> scm_1 | storageType: DISK
> scm_1 | failed: false
> scm_1 | }
> scm_1 | }
> scm_1 | containerReport {
> scm_1 | reports {
> scm_1 | containerID: 1
> scm_1 | used: 1061158912
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1061158912
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 2
> scm_1 | used: 1048576000
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1048576000
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 3
> scm_1 | used: 511705088
> scm_1 | readCount: 0
> scm_1 | writeCount: 32
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 511705088
> scm_1 | state: OPEN
> scm_1 | }
> scm_1 | }
> scm_1 | commandStatusReport {
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerActions {
> scm_1 | containerID: 1
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerID: 2
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | }
> scm_1 |
> scm_1 | --> RPC message response: SCMHeartbeatRequestProto to
> 172.25.0.2:33912
> scm_1 | datanodeUUID: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 |
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:56 -
> Close container Event triggered for container : 1
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:105 -
> container with id : 1 is in CLOSING state and need not be closed.
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:56 -
> Close container Event triggered for container : 2
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:105 -
> container with id : 2 is in CLOSING state and need not be closed.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe,
[jira] [Updated] (HDDS-280) Support ozone dist-start-stitching on openbsd/osx
[
https://issues.apache.org/jira/browse/HDDS-280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-280:
--
Attachment: HDDS-280.003.patch
> Support ozone dist-start-stitching on openbsd/osx
> -
>
> Key: HDDS-280
> URL: https://issues.apache.org/jira/browse/HDDS-280
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-280.001.patch, HDDS-280.002.patch,
> HDDS-280.003.patch
>
>
> {quote}Ozone is creating a symlink during the dist process.
> Using the "ozone" directory as a destination name all the docker-based
> acceptance tests and docker-compose files are more simple as they don't need
> to have the version information in the path.
> But to keep the version specific folder name in the tar file we create a
> symbolic link during the tar creation. With the symbolic link and the
> '–dereference' tar argument we can create the tar file which includes a
> versioned directory (ozone-0.2.1) but we can use the a dist directory without
> the version in the name (hadoop-dist/target/ozone).
> {quote}
> This is the description of the current
> dev-support/bin/ozone-dist-tar-stitching. [~aw] in a comment for HDDS-276
> pointed to the problem that some bsd variants don't support the dereference
> command line options of the ln command.
> The main reason to use this approach is to get a simplified destination name
> without the version (hadoop-dist/target/ozone instead of
> hadoop-dist/target/ozone-0.2.1). It simplifies the docker-compose based
> environments and acceptance tests, therefore I prefer to keep the simplified
> destination name.
> The issue is the tar file creation, if and only if we need the version number
> in the name of the root directory inside of the tar.
> Possible solutions:
> # Use cp target/ozone target/ozone-0.2.1 + tar. It's simple but more slow
> and requires more space.
> # Do the tar distribution from docker all the time in case of 'dereference'
> is not supported. Not very convenient
> # Accept that tar will contain ozone directory and not ozone-0.2.1. This is
> the more simple and can be improved with an additional VERSION file in the
> root of the distribution.
> # (+1) Use hadoop-dist/target/ozone-0.2.1 instead of
> hadoop-dist/target/ozone. This is more complex for the docker based testing
> as we need the explicit names in the compose files (volume:
> ../../../hadoop-dist/target/ozone-0.2.1). The structure is more complex with
> using version in the directory name.
> Please comment your preference.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13882) Change dfs.client.block.write.locateFollowingBlock.retries default from 5 to 10
Kitti Nanasi created HDFS-13882: --- Summary: Change dfs.client.block.write.locateFollowingBlock.retries default from 5 to 10 Key: HDFS-13882 URL: https://issues.apache.org/jira/browse/HDFS-13882 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 3.1.0 Reporter: Kitti Nanasi Assignee: Kitti Nanasi More and more we are seeing cases where customers are running into the java io exception "Unable to close file because the last block does not have enough number of replicas" on client file closure. The common workaround is to increase dfs.client.block.write.locateFollowingBlock.retries from 5 to 10. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-297) Add pipeline actions in Ozone
[ https://issues.apache.org/jira/browse/HDDS-297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-297: --- Attachment: HDDS-297.006.patch > Add pipeline actions in Ozone > - > > Key: HDDS-297 > URL: https://issues.apache.org/jira/browse/HDDS-297 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-297.001.patch, HDDS-297.002.patch, > HDDS-297.003.patch, HDDS-297.004.patch, HDDS-297.005.patch, HDDS-297.006.patch > > > Pipeline in Ozone are created out of a group of nodes depending upon the > replication factor and type. These pipeline provide a transport protocol for > data transfer. > Inorder to detect any failure of pipeline, SCM should receive pipeline > reports from Datanodes and process it to identify various raft rings. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13846) Safe blocks counter is not decremented correctly if the block is striped
[
https://issues.apache.org/jira/browse/HDFS-13846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596141#comment-16596141
]
Kitti Nanasi commented on HDFS-13846:
-
Thanks [~templedf] for the comments! I fixed them in patch v003.
About the javadoc, BlockManagerSafeMode.decrementSafeBlockCount() method is
invoked from BlockManager every time a block is removed, so it is enough to
decrement the counter at the point when it is first less than the "safe"
variable, and what it will do is exactly to decrement the counter when "current
block has fallen below minimal replication".
> Safe blocks counter is not decremented correctly if the block is striped
>
>
> Key: HDFS-13846
> URL: https://issues.apache.org/jira/browse/HDFS-13846
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.0
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Major
> Attachments: HDFS-13846.001.patch, HDFS-13846.002.patch,
> HDFS-13846.003.patch
>
>
> In BlockManagerSafeMode class, the "safe blocks" counter is incremented if
> the number of nodes containing the block equals to the number of data units
> specified by the erasure coding policy, which looks like this in the code:
> {code:java}
> final int safe = storedBlock.isStriped() ?
> ((BlockInfoStriped)storedBlock).getRealDataBlockNum() :
> safeReplication;
> if (storageNum == safe) {
> this.blockSafe++;
> {code}
> But when it is decremented the code does not check if the block is striped or
> not, just compares the number of nodes containing the block with 0
> (safeReplication - 1) if the block is complete, which is not correct.
> {code:java}
> if (storedBlock.isComplete() &&
> blockManager.countNodes(b).liveReplicas() == safeReplication - 1) {
> this.blockSafe--;
> assert blockSafe >= 0;
> checkSafeMode();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13846) Safe blocks counter is not decremented correctly if the block is striped
[
https://issues.apache.org/jira/browse/HDFS-13846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kitti Nanasi updated HDFS-13846:
Attachment: HDFS-13846.003.patch
> Safe blocks counter is not decremented correctly if the block is striped
>
>
> Key: HDFS-13846
> URL: https://issues.apache.org/jira/browse/HDFS-13846
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.0
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Major
> Attachments: HDFS-13846.001.patch, HDFS-13846.002.patch,
> HDFS-13846.003.patch
>
>
> In BlockManagerSafeMode class, the "safe blocks" counter is incremented if
> the number of nodes containing the block equals to the number of data units
> specified by the erasure coding policy, which looks like this in the code:
> {code:java}
> final int safe = storedBlock.isStriped() ?
> ((BlockInfoStriped)storedBlock).getRealDataBlockNum() :
> safeReplication;
> if (storageNum == safe) {
> this.blockSafe++;
> {code}
> But when it is decremented the code does not check if the block is striped or
> not, just compares the number of nodes containing the block with 0
> (safeReplication - 1) if the block is complete, which is not correct.
> {code:java}
> if (storedBlock.isComplete() &&
> blockManager.countNodes(b).liveReplicas() == safeReplication - 1) {
> this.blockSafe--;
> assert blockSafe >= 0;
> checkSafeMode();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13859) Add update replicaInfo's volume in LocalReplica#updateWithReplica
[
https://issues.apache.org/jira/browse/HDFS-13859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liaoyuxiangqin updated HDFS-13859:
--
Status: Patch Available (was: Open)
> Add update replicaInfo's volume in LocalReplica#updateWithReplica
> --
>
> Key: HDFS-13859
> URL: https://issues.apache.org/jira/browse/HDFS-13859
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Major
> Attachments: HDFS-13859.001.patch
>
>
> When DirectoryScanner used diff ScanInfo to check and update with
> memBlock, i found the LocalReplica#updateWithReplica only update the diskfile
> path but without replicaInfo's volume. And may be the memblock ' volume is
> diff with the diskfile path before directory scan, so i think need to update
> the volume meanwhile,so the replicaInfo's meminfo is consistent with disk
> storage。The relation code as follows:
> {code:java}
> public void updateWithReplica(StorageLocation replicaLocation) {
> // for local replicas, the replica location is assumed to be a file.
> File diskFile = null;
> try {
> diskFile = new File(replicaLocation.getUri());
> } catch (IllegalArgumentException e) {
> diskFile = null;
> }
> if (null == diskFile) {
> setDirInternal(null);
> } else {
> setDirInternal(diskFile.getParentFile());
> }
> }
> {code}
> Thanks all!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13859) Add update replicaInfo's volume in LocalReplica#updateWithReplica
[
https://issues.apache.org/jira/browse/HDFS-13859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liaoyuxiangqin updated HDFS-13859:
--
Attachment: HDFS-13859.001.patch
> Add update replicaInfo's volume in LocalReplica#updateWithReplica
> --
>
> Key: HDFS-13859
> URL: https://issues.apache.org/jira/browse/HDFS-13859
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Major
> Attachments: HDFS-13859.001.patch
>
>
> When DirectoryScanner used diff ScanInfo to check and update with
> memBlock, i found the LocalReplica#updateWithReplica only update the diskfile
> path but without replicaInfo's volume. And may be the memblock ' volume is
> diff with the diskfile path before directory scan, so i think need to update
> the volume meanwhile,so the replicaInfo's meminfo is consistent with disk
> storage。The relation code as follows:
> {code:java}
> public void updateWithReplica(StorageLocation replicaLocation) {
> // for local replicas, the replica location is assumed to be a file.
> File diskFile = null;
> try {
> diskFile = new File(replicaLocation.getUri());
> } catch (IllegalArgumentException e) {
> diskFile = null;
> }
> if (null == diskFile) {
> setDirInternal(null);
> } else {
> setDirInternal(diskFile.getParentFile());
> }
> }
> {code}
> Thanks all!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13634) RBF: Configurable value in xml for async connection request queue size.
[
https://issues.apache.org/jira/browse/HDFS-13634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-13634:
-
Affects Version/s: 3.1.0
2.9.1
3.0.3
> RBF: Configurable value in xml for async connection request queue size.
> ---
>
> Key: HDFS-13634
> URL: https://issues.apache.org/jira/browse/HDFS-13634
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.1.0, 2.9.1, 3.0.3
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 3.1.2
>
> Attachments: HDFS-13634.0.patch, HDFS-13634.1.patch,
> HDFS-13634.2.patch, HDFS-13634.3.patch
>
>
> The below in ConnectionManager.java should be configurable via hdfs-site.xml.
> This a very critical parameter for routers, admins would like to change this
> without doing a new build.
> {code:java}
> /** Number of parallel new connections to create. */
> protected static final int MAX_NEW_CONNECTIONS = 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13634) RBF: Configurable value in xml for async connection request queue size.
[
https://issues.apache.org/jira/browse/HDFS-13634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-13634:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.2
3.2.0
2.10.0
Status: Resolved (was: Patch Available)
Thanks [~crh] for the contribution and thanks others for the reviews.
> RBF: Configurable value in xml for async connection request queue size.
> ---
>
> Key: HDFS-13634
> URL: https://issues.apache.org/jira/browse/HDFS-13634
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 3.1.2
>
> Attachments: HDFS-13634.0.patch, HDFS-13634.1.patch,
> HDFS-13634.2.patch, HDFS-13634.3.patch
>
>
> The below in ConnectionManager.java should be configurable via hdfs-site.xml.
> This a very critical parameter for routers, admins would like to change this
> without doing a new build.
> {code:java}
> /** Number of parallel new connections to create. */
> protected static final int MAX_NEW_CONNECTIONS = 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-75) Ozone: Support CopyContainer
[ https://issues.apache.org/jira/browse/HDDS-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596063#comment-16596063 ] Elek, Marton commented on HDDS-75: -- The patch is rebased and updated. Checkstyle/findbugs/unit test issues are fixed (hopefully...). > Ozone: Support CopyContainer > > > Key: HDDS-75 > URL: https://issues.apache.org/jira/browse/HDDS-75 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Anu Engineer >Assignee: Elek, Marton >Priority: Blocker > Fix For: 0.2.1 > > Attachments: HDDS-75.005.patch, HDDS-75.006.patch, HDDS-75.007.patch, > HDDS-75.009.patch, HDDS-75.010.patch, HDDS-75.011.patch, HDDS-75.012.patch, > HDFS-11686-HDFS-7240.001.patch, HDFS-11686-HDFS-7240.002.patch, > HDFS-11686-HDFS-7240.003.patch, HDFS-11686-HDFS-7240.004.patch > > > Once a container is closed we need to copy the container to the correct pool > or re-encode the container to use erasure coding. The copyContainer allows > users to get the container as a tarball from the remote machine. > The copyContainer is a basic step to move the raw container data from one > datanode to an other node. It could be used by higher level components such > like the scm which ensures that the replication rules are satisfied. > The CopyContainer by default works in pull model: the destination datanode > could read the raw data from one or more source datanode where the container > exists. > The source provides a binary representation of the container over a common > interface which has two method: > # prepare(containerName) > # copyData(String containerName, OutputStream destination) > Prepare phase is called right after the closing event and the implementation > could prepare for the copy by precreate a compressed tar file from the > container data. As a first step we can provide a simple implementation which > creates the tar files on demand. > The destination datanode should retry the copy if the container in the source > node not yet prepared. > The raw container data is provided over HTTP. The HTTP endpoint should be > separated from the ObjectStore REST API (similar to the distinctions between > HDFS-7240 and HDFS-13074) > Long-term the HTTP endpoint should support Http-Range requests: One container > could be copied from multiple source by the destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-75) Ozone: Support CopyContainer
[ https://issues.apache.org/jira/browse/HDDS-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-75: - Attachment: HDDS-75.012.patch > Ozone: Support CopyContainer > > > Key: HDDS-75 > URL: https://issues.apache.org/jira/browse/HDDS-75 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Anu Engineer >Assignee: Elek, Marton >Priority: Blocker > Fix For: 0.2.1 > > Attachments: HDDS-75.005.patch, HDDS-75.006.patch, HDDS-75.007.patch, > HDDS-75.009.patch, HDDS-75.010.patch, HDDS-75.011.patch, HDDS-75.012.patch, > HDFS-11686-HDFS-7240.001.patch, HDFS-11686-HDFS-7240.002.patch, > HDFS-11686-HDFS-7240.003.patch, HDFS-11686-HDFS-7240.004.patch > > > Once a container is closed we need to copy the container to the correct pool > or re-encode the container to use erasure coding. The copyContainer allows > users to get the container as a tarball from the remote machine. > The copyContainer is a basic step to move the raw container data from one > datanode to an other node. It could be used by higher level components such > like the scm which ensures that the replication rules are satisfied. > The CopyContainer by default works in pull model: the destination datanode > could read the raw data from one or more source datanode where the container > exists. > The source provides a binary representation of the container over a common > interface which has two method: > # prepare(containerName) > # copyData(String containerName, OutputStream destination) > Prepare phase is called right after the closing event and the implementation > could prepare for the copy by precreate a compressed tar file from the > container data. As a first step we can provide a simple implementation which > creates the tar files on demand. > The destination datanode should retry the copy if the container in the source > node not yet prepared. > The raw container data is provided over HTTP. The HTTP endpoint should be > separated from the ObjectStore REST API (similar to the distinctions between > HDFS-7240 and HDFS-13074) > Long-term the HTTP endpoint should support Http-Range requests: One container > could be copied from multiple source by the destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-380) Remove synchronization from ChunkGroupOutputStream and ChunkOutputStream
[ https://issues.apache.org/jira/browse/HDDS-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596055#comment-16596055 ] Nanda kumar commented on HDDS-380: -- Thanks [~shashikant] for the contribution. I have committed this to trunk. > Remove synchronization from ChunkGroupOutputStream and ChunkOutputStream > > > Key: HDDS-380 > URL: https://issues.apache.org/jira/browse/HDDS-380 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-380.00.patch > > > In one of the code review for HDDS-247, it was suggested that > ChunkGroupOutputStream/ChunkOutputStream may not be thread safe as the Java > OutputStream subclasses are generally non-thread-safe for performance > reasons. If users want thread safe, they can easily synchronize it > themselves. This Jira aims to remove synchronization from > ChunkGroupOutputStream and ChunkOutputStream. > cc [~szetszwo]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-380) Remove synchronization from ChunkGroupOutputStream and ChunkOutputStream
[ https://issues.apache.org/jira/browse/HDDS-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar updated HDDS-380: - Resolution: Fixed Status: Resolved (was: Patch Available) > Remove synchronization from ChunkGroupOutputStream and ChunkOutputStream > > > Key: HDDS-380 > URL: https://issues.apache.org/jira/browse/HDDS-380 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-380.00.patch > > > In one of the code review for HDDS-247, it was suggested that > ChunkGroupOutputStream/ChunkOutputStream may not be thread safe as the Java > OutputStream subclasses are generally non-thread-safe for performance > reasons. If users want thread safe, they can easily synchronize it > themselves. This Jira aims to remove synchronization from > ChunkGroupOutputStream and ChunkOutputStream. > cc [~szetszwo]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13634) RBF: Configurable value in xml for async connection request queue size.
[
https://issues.apache.org/jira/browse/HDFS-13634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596053#comment-16596053
]
Yiqun Lin commented on HDFS-13634:
--
+1. I'd like to commit this shortly, :).
> RBF: Configurable value in xml for async connection request queue size.
> ---
>
> Key: HDFS-13634
> URL: https://issues.apache.org/jira/browse/HDFS-13634
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-13634.0.patch, HDFS-13634.1.patch,
> HDFS-13634.2.patch, HDFS-13634.3.patch
>
>
> The below in ConnectionManager.java should be configurable via hdfs-site.xml.
> This a very critical parameter for routers, admins would like to change this
> without doing a new build.
> {code:java}
> /** Number of parallel new connections to create. */
> protected static final int MAX_NEW_CONNECTIONS = 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13774) EC: "hdfs ec -getPolicy" is not retrieving policy details when the special REPLICATION policy set on the directory
[ https://issues.apache.org/jira/browse/HDFS-13774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596050#comment-16596050 ] Ayush Saxena commented on HDFS-13774: - [~vinayrpet] can you pls review!!! :) > EC: "hdfs ec -getPolicy" is not retrieving policy details when the special > REPLICATION policy set on the directory > -- > > Key: HDFS-13774 > URL: https://issues.apache.org/jira/browse/HDFS-13774 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 > Environment: 3 Node Linux Cluster >Reporter: Souryakanta Dwivedy >Assignee: Ayush Saxena >Priority: Minor > Attachments: GetPolicy_EC.png, HDFS-13774-01.patch > > > Erasure coding: "hdfs ec -getPolicy"" is not retrieving policy details when > the special REPLICATION policy set on the directory > Steps :- > - Create a directory "testEC" > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > - Enable any Erasure coding policy like "XOR-2-1-1024k" > - Set the EC Policy on the Directory > - Get the EC policy for the directory [Received message as : "XOR-2-1-1024k" ] > - Now again set the EC Policy on the directory as "replicate" special > REPLICATION policy > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > The policy is being set for the Directory ,but while retrieving policy > details its throwing error as > policy for the directory is unspecified which is wrong behavior -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-380) Remove synchronization from ChunkGroupOutputStream and ChunkOutputStream
[ https://issues.apache.org/jira/browse/HDDS-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596049#comment-16596049 ] Nanda kumar commented on HDDS-380: -- Thanks [~shashikant] for the patch. +1, looks good to me. I will commit this shortly. > Remove synchronization from ChunkGroupOutputStream and ChunkOutputStream > > > Key: HDDS-380 > URL: https://issues.apache.org/jira/browse/HDDS-380 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-380.00.patch > > > In one of the code review for HDDS-247, it was suggested that > ChunkGroupOutputStream/ChunkOutputStream may not be thread safe as the Java > OutputStream subclasses are generally non-thread-safe for performance > reasons. If users want thread safe, they can easily synchronize it > themselves. This Jira aims to remove synchronization from > ChunkGroupOutputStream and ChunkOutputStream. > cc [~szetszwo]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13863) FsDatasetImpl should log DiskOutOfSpaceException
[
https://issues.apache.org/jira/browse/HDFS-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596047#comment-16596047
]
Yiqun Lin edited comment on HDFS-13863 at 8/29/18 7:46 AM:
---
LGTM, +1. Hold off the commit to let [~arpitagarwal] to have a look.
was (Author: linyiqun):
LGMT, +1. Hold off the commit to let [~arpitagarwal] to have a look.
> FsDatasetImpl should log DiskOutOfSpaceException
>
>
> Key: HDFS-13863
> URL: https://issues.apache.org/jira/browse/HDFS-13863
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.0, 2.9.1, 3.0.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-13863.001.patch, HDFS-13863.002.patch,
> HDFS-13863.003.patch
>
>
> The code in function *createRbw* as follow
> {code:java}
> try {
> // First try to place the block on a transient volume.
> ref = volumes.getNextTransientVolume(b.getNumBytes());
> datanode.getMetrics().incrRamDiskBlocksWrite();
> } catch (DiskOutOfSpaceException de) {
> // Ignore the exception since we just fall back to persistent
> storage.
> } finally {
> if (ref == null) {
> cacheManager.release(b.getNumBytes());
> }
> }
> {code}
> I think we should log the exception because it took me long time to resolve
> problems, and maybe others face the same problems.
> When i test ram_disk, i found no data was written into randomdisk. I debug,
> deep into the source code, and found that randomdisk size was less than
> reserved space. I think if message was logged, i would resolve the problem
> quickly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13863) FsDatasetImpl should log DiskOutOfSpaceException
[
https://issues.apache.org/jira/browse/HDFS-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596047#comment-16596047
]
Yiqun Lin commented on HDFS-13863:
--
LGMT, +1. Hold off the commit to let [~arpitagarwal] to have a look.
> FsDatasetImpl should log DiskOutOfSpaceException
>
>
> Key: HDFS-13863
> URL: https://issues.apache.org/jira/browse/HDFS-13863
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.0, 2.9.1, 3.0.3
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-13863.001.patch, HDFS-13863.002.patch,
> HDFS-13863.003.patch
>
>
> The code in function *createRbw* as follow
> {code:java}
> try {
> // First try to place the block on a transient volume.
> ref = volumes.getNextTransientVolume(b.getNumBytes());
> datanode.getMetrics().incrRamDiskBlocksWrite();
> } catch (DiskOutOfSpaceException de) {
> // Ignore the exception since we just fall back to persistent
> storage.
> } finally {
> if (ref == null) {
> cacheManager.release(b.getNumBytes());
> }
> }
> {code}
> I think we should log the exception because it took me long time to resolve
> problems, and maybe others face the same problems.
> When i test ram_disk, i found no data was written into randomdisk. I debug,
> deep into the source code, and found that randomdisk size was less than
> reserved space. I think if message was logged, i would resolve the problem
> quickly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-348) Parallalize container ops in ContainerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-348: - Description: Currently all the ops in ContainerStateMachine#applyTransaction are processed sequentially by the ContainerStateMachine. However these ops can be parallelized by having a configurable no of executors where a certain executor for a container can be chosen from the pool of executors by hashing the containerID. (was: Currently all the ops in ContainerStateMachine#applyTransaction are processed sequentially by the ContainerStateMachine. However these ops can be parallelized by having a per container executor. ) > Parallalize container ops in ContainerStateMachine > -- > > Key: HDDS-348 > URL: https://issues.apache.org/jira/browse/HDDS-348 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.2.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-348.001.patch, HDDS-348.002.patch > > > Currently all the ops in ContainerStateMachine#applyTransaction are processed > sequentially by the ContainerStateMachine. However these ops can be > parallelized by having a configurable no of executors where a certain > executor for a container can be chosen from the pool of executors by hashing > the containerID. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-348) Parallalize container ops in ContainerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596045#comment-16596045 ] Shashikant Banerjee commented on HDDS-348: -- Thanks [~msingh], for reporting and working on this issue. The patch overall looks good to me. # When stateMachineHelper is instantiated, we can assign the executor to it right away as it only handles operations for a particular container. That way , it will not be required to find which executor it needs to execute the commands in during execution. # We can remove writeChunkExecutor and use the container specific executor added with the patch to perform write chunk operations . # XceiverServerRatis.java : L140-144 : Instead of of defining a singkle threaded executor instance we can have a configurable no of threads per executor. Default value may be set to 1. # ScmConfigKeys.java: 60 : "dfs.container.ratis.num.container.op.threads" is actually the no of executors being used rather than no of threads. The config name should be changed accordingly. Same thing applies to changes in OzoneConfigKeys.java. # It would be good to add some more comments in ContainerStateMachine class regarding work flow with executors. > Parallalize container ops in ContainerStateMachine > -- > > Key: HDDS-348 > URL: https://issues.apache.org/jira/browse/HDDS-348 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.2.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-348.001.patch, HDDS-348.002.patch > > > Currently all the ops in ContainerStateMachine#applyTransaction are processed > sequentially by the ContainerStateMachine. However these ops can be > parallelized by having a configurable no of executors where a certain > executor for a container can be chosen from the pool of executors by hashing > the containerID. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-336) Print out container location information for a specific ozone key
[ https://issues.apache.org/jira/browse/HDDS-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596041#comment-16596041 ] LiXin Ge commented on HDDS-336: --- Failed junit tests passed in my local machine. I will fix the checkstyle warnnings when jenkins resumed, the checkstyle report link can't be opened all the day. > Print out container location information for a specific ozone key > -- > > Key: HDDS-336 > URL: https://issues.apache.org/jira/browse/HDDS-336 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: LiXin Ge >Priority: Major > Labels: newbie > Fix For: 0.2.1 > > Attachments: HDDS-336.000.patch, HDDS-336.001.patch, > HDDS-336.002.patch, HDDS-336.003.patch > > > In the protobuf protocol we have all the containerid/localid(=blockid) > information for a specific ozone key. > It would be a big help to print out this information to the command line with > the ozone cli. > It requires to improve the REST and RPC interface with additionalOzone > KeyLocation information. > It would help a very big help during the test of the current scm behaviour. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13815) RBF: Add check to order command
[ https://issues.apache.org/jira/browse/HDFS-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596037#comment-16596037 ] Ranith Sardar commented on HDFS-13815: -- [~brahmareddy], Updated the patch to specify the printUsage msg. > RBF: Add check to order command > --- > > Key: HDFS-13815 > URL: https://issues.apache.org/jira/browse/HDFS-13815 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation >Affects Versions: 3.0.0 >Reporter: Soumyapn >Assignee: Ranith Sardar >Priority: Major > Attachments: HDFS-13815-001.patch, HDFS-13815-002.patch, > HDFS-13815-003.patch, HDFS-13815-004.patch, HDFS-13815-005.patch, > HDFS-13815-006.patch > > > No check being done on order command. > It says successfully updated mount table if we don't specify order command > and it is not updated in mount table > Execute the dfsrouter update command with the below scenarios. > 1. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 RANDOM > 2. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 -or RANDOM > 3. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 -ord RANDOM > 4. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 -orde RANDOM > > The console message says, Successfully updated mount point. But it is not > updated in the mount table. > > Expected Result: > Exception on console as the order command is missing/not written properl -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13815) RBF: Add check to order command
[ https://issues.apache.org/jira/browse/HDFS-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ranith Sardar updated HDFS-13815: - Attachment: HDFS-13815-006.patch > RBF: Add check to order command > --- > > Key: HDFS-13815 > URL: https://issues.apache.org/jira/browse/HDFS-13815 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation >Affects Versions: 3.0.0 >Reporter: Soumyapn >Assignee: Ranith Sardar >Priority: Major > Attachments: HDFS-13815-001.patch, HDFS-13815-002.patch, > HDFS-13815-003.patch, HDFS-13815-004.patch, HDFS-13815-005.patch, > HDFS-13815-006.patch > > > No check being done on order command. > It says successfully updated mount table if we don't specify order command > and it is not updated in mount table > Execute the dfsrouter update command with the below scenarios. > 1. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 RANDOM > 2. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 -or RANDOM > 3. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 -ord RANDOM > 4. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 -orde RANDOM > > The console message says, Successfully updated mount point. But it is not > updated in the mount table. > > Expected Result: > Exception on console as the order command is missing/not written properl -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-379) Simplify and improve the cli arg parsing of ozone scmcli
[ https://issues.apache.org/jira/browse/HDDS-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596034#comment-16596034 ] Anu Engineer commented on HDDS-379: --- [~elek] Findbugs? > Simplify and improve the cli arg parsing of ozone scmcli > > > Key: HDDS-379 > URL: https://issues.apache.org/jira/browse/HDDS-379 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-379.001.patch > > > SCMCLI is a useful tool to test SCM. It can create/delete/close/list > containers. > There are multiple problems with the current scmcli. > The biggest one is the cli argument handling. Similar to HDDS-190, it's often > very hard to get the help for a specific subcommand. > The other one is that a big part of the code is the argument handling which > is mixed with the business logic. > I propose to use a more modern argument handler library and simplify the > argument handling (and improve the user experience). > I propose to use [picocli|https://github.com/remkop/picocli]. > 1.) It supports subcommands and subcommand specific and general arguments. > 2.) It could work based on annotation with very few additional boilerplate > code > 3.) It's very well documented and easy to use > 4.) It's licenced under Apache licence > 5.) It supports tab autocompletion for bash and zsh and colorful output > 6.) Actively maintainer project > 7.) Adopter by other bigger projects (groovy, junit, log4j) > In this patch I would like to demonstrate how the cli handling could be > simplified. And if it's accepted, we can start to use similar approach for > other ozone cli as well. > The patch also fixes the cli (the name of the main class was wrong). > It also requires HDDS-377 for the be compiled. > I also deleted the TestSCMCli. It was turned off with an annotation and I > believe that this functionality could be tested more easily with a robot test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-280) Support ozone dist-start-stitching on openbsd/osx
[
https://issues.apache.org/jira/browse/HDDS-280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596032#comment-16596032
]
Anu Engineer commented on HDDS-280:
---
[~aw],[~elek] I will commit this shortly. Thanks for the reviews and
contribution.
> Support ozone dist-start-stitching on openbsd/osx
> -
>
> Key: HDDS-280
> URL: https://issues.apache.org/jira/browse/HDDS-280
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-280.001.patch, HDDS-280.002.patch
>
>
> {quote}Ozone is creating a symlink during the dist process.
> Using the "ozone" directory as a destination name all the docker-based
> acceptance tests and docker-compose files are more simple as they don't need
> to have the version information in the path.
> But to keep the version specific folder name in the tar file we create a
> symbolic link during the tar creation. With the symbolic link and the
> '–dereference' tar argument we can create the tar file which includes a
> versioned directory (ozone-0.2.1) but we can use the a dist directory without
> the version in the name (hadoop-dist/target/ozone).
> {quote}
> This is the description of the current
> dev-support/bin/ozone-dist-tar-stitching. [~aw] in a comment for HDDS-276
> pointed to the problem that some bsd variants don't support the dereference
> command line options of the ln command.
> The main reason to use this approach is to get a simplified destination name
> without the version (hadoop-dist/target/ozone instead of
> hadoop-dist/target/ozone-0.2.1). It simplifies the docker-compose based
> environments and acceptance tests, therefore I prefer to keep the simplified
> destination name.
> The issue is the tar file creation, if and only if we need the version number
> in the name of the root directory inside of the tar.
> Possible solutions:
> # Use cp target/ozone target/ozone-0.2.1 + tar. It's simple but more slow
> and requires more space.
> # Do the tar distribution from docker all the time in case of 'dereference'
> is not supported. Not very convenient
> # Accept that tar will contain ozone directory and not ozone-0.2.1. This is
> the more simple and can be improved with an additional VERSION file in the
> root of the distribution.
> # (+1) Use hadoop-dist/target/ozone-0.2.1 instead of
> hadoop-dist/target/ozone. This is more complex for the docker based testing
> as we need the explicit names in the compose files (volume:
> ../../../hadoop-dist/target/ozone-0.2.1). The structure is more complex with
> using version in the directory name.
> Please comment your preference.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-373) Ozone genconf tool must generate ozone-site.xml with sample values instead of a template
[ https://issues.apache.org/jira/browse/HDDS-373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-373: -- Fix Version/s: (was: 0.2.1) 0.3.0 > Ozone genconf tool must generate ozone-site.xml with sample values instead of > a template > > > Key: HDDS-373 > URL: https://issues.apache.org/jira/browse/HDDS-373 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Tools >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > Fix For: 0.3.0 > > Attachments: HDDS-373.001.patch > > > As discussed with [~anu], currently, the genconf tool generates a template > ozone-site.xml. This is not very useful for new users as they would have to > understand what values should be set for the minimal configuration properties. > This Jira proposes to modify the ozone-default.xml which is leveraged by > genconf tool to generate ozone-site.xml -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-348) Parallalize container ops in ContainerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-348: --- Attachment: HDDS-348.002.patch > Parallalize container ops in ContainerStateMachine > -- > > Key: HDDS-348 > URL: https://issues.apache.org/jira/browse/HDDS-348 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.2.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-348.001.patch, HDDS-348.002.patch > > > Currently all the ops in ContainerStateMachine#applyTransaction are processed > sequentially by the ContainerStateMachine. However these ops can be > parallelized by having a per container executor. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13815) RBF: Add check to order command
[
https://issues.apache.org/jira/browse/HDFS-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595992#comment-16595992
]
Brahma Reddy Battula commented on HDFS-13815:
-
bq. Would you mind if i solve this in HDFS-13842 (as this is also related to
addMount).
ok.
Apart from following, patch lgtm.
how about changing *printUsage()* to *printUsage("add"), printUsage("update")*,
so that relevant help will be logged. [~linyiqun] what do you think.?
> RBF: Add check to order command
> ---
>
> Key: HDFS-13815
> URL: https://issues.apache.org/jira/browse/HDFS-13815
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.0.0
>Reporter: Soumyapn
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-13815-001.patch, HDFS-13815-002.patch,
> HDFS-13815-003.patch, HDFS-13815-004.patch, HDFS-13815-005.patch
>
>
> No check being done on order command.
> It says successfully updated mount table if we don't specify order command
> and it is not updated in mount table
> Execute the dfsrouter update command with the below scenarios.
> 1. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 RANDOM
> 2. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 -or RANDOM
> 3. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 -ord RANDOM
> 4. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 -orde RANDOM
>
> The console message says, Successfully updated mount point. But it is not
> updated in the mount table.
>
> Expected Result:
> Exception on console as the order command is missing/not written properl
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13815) RBF: Add check to order command
[ https://issues.apache.org/jira/browse/HDFS-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595975#comment-16595975 ] Yiqun Lin commented on HDFS-13815: -- LGTM, +1 pending Jenkins. [~brahmareddy], I also agree do the refactor work in HDFS-13842 and i merge this soon. > RBF: Add check to order command > --- > > Key: HDFS-13815 > URL: https://issues.apache.org/jira/browse/HDFS-13815 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation >Affects Versions: 3.0.0 >Reporter: Soumyapn >Assignee: Ranith Sardar >Priority: Major > Attachments: HDFS-13815-001.patch, HDFS-13815-002.patch, > HDFS-13815-003.patch, HDFS-13815-004.patch, HDFS-13815-005.patch > > > No check being done on order command. > It says successfully updated mount table if we don't specify order command > and it is not updated in mount table > Execute the dfsrouter update command with the below scenarios. > 1. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 RANDOM > 2. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 -or RANDOM > 3. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 -ord RANDOM > 4. ./hdfs dfsrouteradmin -update /apps3 hacluster,ns2 /tmp6 -orde RANDOM > > The console message says, Successfully updated mount point. But it is not > updated in the mount table. > > Expected Result: > Exception on console as the order command is missing/not written properl -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
