[
https://issues.apache.org/jira/browse/HDFS-16978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720665#comment-17720665
]
ASF GitHub Bot commented on HDFS-16978:
---------------------------------------
virajjasani commented on code in PR #5554:
URL: https://github.com/apache/hadoop/pull/5554#discussion_r1187807675
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java:
##########
@@ -462,6 +482,136 @@ public int run(String[] argv) throws Exception {
return exitCode;
}
+ /**
+ * Add all mount point entries provided in the request.
+ *
+ * @param parameters Parameters for the mount points.
+ * @param i Current index on the parameters array.
+ * @return True if adding all mount points was successful, False otherwise.
+ * @throws IOException If the RPC call to add the mount points fail.
+ */
+ private boolean addAllMount(String[] parameters, int i) throws IOException {
+ List<AddMountAttributes> addMountAttributesList = new ArrayList<>();
+ while (i < parameters.length) {
+ AddMountAttributes addMountAttributes =
getAddMountAttributes(parameters, i, true);
+ if (addMountAttributes == null) {
+ return false;
+ }
+ i = addMountAttributes.getParamIndex();
+ addMountAttributesList.add(addMountAttributes);
+ }
+ List<MountTable> addEntries =
getMountTablesFromAddAllAttributes(addMountAttributesList);
+ AddMountTableEntriesRequest request =
+ AddMountTableEntriesRequest.newInstance(addEntries);
+ MountTableManager mountTable = client.getMountTableManager();
+ AddMountTableEntriesResponse addResponse =
+ mountTable.addMountTableEntries(request);
+ boolean added = addResponse.getStatus();
+ if (!added) {
+ System.err.println("Cannot add some or all mount points");
+ }
Review Comment:
Good point!
For admin impl, it is not a big deal to cherry-pick on what response state
store APIs support, however, as of today, putAll() at state store level does
not support returning failed entries. we can take this up as a follow-up PR, so
that this PR does not have to deal with all state store impl level changes
(just to keep it clean). hence, let me take this up as a follow-up jira, sounds
good to you?
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java:
##########
@@ -462,6 +482,136 @@ public int run(String[] argv) throws Exception {
return exitCode;
}
+ /**
+ * Add all mount point entries provided in the request.
+ *
+ * @param parameters Parameters for the mount points.
+ * @param i Current index on the parameters array.
+ * @return True if adding all mount points was successful, False otherwise.
+ * @throws IOException If the RPC call to add the mount points fail.
+ */
+ private boolean addAllMount(String[] parameters, int i) throws IOException {
+ List<AddMountAttributes> addMountAttributesList = new ArrayList<>();
+ while (i < parameters.length) {
+ AddMountAttributes addMountAttributes =
getAddMountAttributes(parameters, i, true);
+ if (addMountAttributes == null) {
+ return false;
+ }
+ i = addMountAttributes.getParamIndex();
+ addMountAttributesList.add(addMountAttributes);
+ }
+ List<MountTable> addEntries =
getMountTablesFromAddAllAttributes(addMountAttributesList);
+ AddMountTableEntriesRequest request =
+ AddMountTableEntriesRequest.newInstance(addEntries);
+ MountTableManager mountTable = client.getMountTableManager();
+ AddMountTableEntriesResponse addResponse =
+ mountTable.addMountTableEntries(request);
+ boolean added = addResponse.getStatus();
+ if (!added) {
+ System.err.println("Cannot add some or all mount points");
+ }
+ return added;
+ }
+
+ /**
+ * From the given params, form and retrieve AddMountAttributes object. This
object is meant
+ * to be used while adding single or multiple mount points with their own
specific attributes.
+ *
+ * @param parameters Parameters for the mount point.
+ * @param i Current index on the parameters array.
+ * @param isMultipleAdd True if multiple mount points are to be added, False
if single mount
+ * point is to be added.
+ * @return AddMountAttributes object.
+ */
+ private static AddMountAttributes getAddMountAttributes(String[] parameters,
int i,
+ boolean isMultipleAdd) {
+ // Mandatory parameters
+ String mount = parameters[i++];
+ String[] nss = parameters[i++].split(",");
+ String dest = parameters[i++];
Review Comment:
makes sense, but we already support same dest nsid for multiple dest path
(ns0->/path1 and ns1->/path1) as part of `-add` right? hence i was trying to
keep that behavior for `-addAll` as well.
when we update single add behavior to support comma separated dest path as
well (ns0->/path1 and ns1->/path2), at that time we can make change here to
make it applicable for both, wdyt?
also, for single add, if we provide multiple nsid and same dest path, it
still internally results into single add entry to mount table. it's just that
when we finally start supporting multiple nsid and multiple dest path as part
of single add command, making change to this area might be more impactful?
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java:
##########
@@ -462,6 +482,136 @@ public int run(String[] argv) throws Exception {
return exitCode;
}
+ /**
+ * Add all mount point entries provided in the request.
+ *
+ * @param parameters Parameters for the mount points.
+ * @param i Current index on the parameters array.
+ * @return True if adding all mount points was successful, False otherwise.
+ * @throws IOException If the RPC call to add the mount points fail.
+ */
+ private boolean addAllMount(String[] parameters, int i) throws IOException {
+ List<AddMountAttributes> addMountAttributesList = new ArrayList<>();
+ while (i < parameters.length) {
+ AddMountAttributes addMountAttributes =
getAddMountAttributes(parameters, i, true);
+ if (addMountAttributes == null) {
+ return false;
+ }
+ i = addMountAttributes.getParamIndex();
+ addMountAttributesList.add(addMountAttributes);
+ }
+ List<MountTable> addEntries =
getMountTablesFromAddAllAttributes(addMountAttributesList);
+ AddMountTableEntriesRequest request =
+ AddMountTableEntriesRequest.newInstance(addEntries);
+ MountTableManager mountTable = client.getMountTableManager();
+ AddMountTableEntriesResponse addResponse =
+ mountTable.addMountTableEntries(request);
+ boolean added = addResponse.getStatus();
+ if (!added) {
+ System.err.println("Cannot add some or all mount points");
+ }
+ return added;
+ }
+
+ /**
+ * From the given params, form and retrieve AddMountAttributes object. This
object is meant
+ * to be used while adding single or multiple mount points with their own
specific attributes.
+ *
+ * @param parameters Parameters for the mount point.
+ * @param i Current index on the parameters array.
+ * @param isMultipleAdd True if multiple mount points are to be added, False
if single mount
+ * point is to be added.
+ * @return AddMountAttributes object.
+ */
+ private static AddMountAttributes getAddMountAttributes(String[] parameters,
int i,
+ boolean isMultipleAdd) {
+ // Mandatory parameters
+ String mount = parameters[i++];
+ String[] nss = parameters[i++].split(",");
+ String dest = parameters[i++];
Review Comment:
At high level, `dest.length != 1` still seems good for `-addAll` but perhaps
restricting it with `dest.length !=nss.length` is maybe not needed as such IMO
> RBF: Admin command to support bulk add of mount points
> ------------------------------------------------------
>
> Key: HDFS-16978
> URL: https://issues.apache.org/jira/browse/HDFS-16978
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Minor
> Labels: pull-request-available
>
> All state store implementations support adding multiple state store records
> using single putAll() implementation. We should provide new router admin API
> to support bulk addition of mount table entries that can utilize this build
> add implementation at state store level.
> For more than one mount point to be added, the goal of bulk addition should be
> # To reduce frequent router calls
> # To avoid frequent state store cache refreshers with each single mount
> point addition
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
