[
https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583851#comment-17583851
]
ASF GitHub Bot commented on YARN-9708:
--------------------------------------
goiri commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953096007
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/YARNDelegationTokenIdentifier.java:
##########
@@ -112,4 +115,14 @@ public YARNDelegationTokenIdentifierProto getProto() {
setBuilderFields();
return builder.build();
}
+
+ @InterfaceAudience.Private
Review Comment:
Can we import the InterfaceAudience.Private as others?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterRMDTSecretManagerState.java:
##########
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.federation.store.records;
+
+import org.apache.hadoop.security.token.delegation.DelegationKey;
+import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+public class RouterRMDTSecretManagerState {
+
+ // DTIdentifier -> renewDate
+ private Map<RMDelegationTokenIdentifier, Long> delegationTokenState =
+ new HashMap<RMDelegationTokenIdentifier, Long>();
Review Comment:
`new HashMap<>();` and single line
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterRMTokenRequest.java:
##########
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.federation.store.records;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.yarn.util.Records;
+
[email protected]
[email protected]
+public abstract class RouterRMTokenRequest {
+
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public static RouterRMTokenRequest newInstance(RouterStoreToken
routerStoreToken) {
+ RouterRMTokenRequest request =
Records.newRecord(RouterRMTokenRequest.class);
+ request.setRouterStoreToken(routerStoreToken);
+ return request;
+ }
+
+ @InterfaceAudience.Public
+ @InterfaceStability.Unstable
+ public abstract RouterStoreToken getRouterStoreToken();
+
+ @InterfaceAudience.Private
Review Comment:
Import statically this.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/secure/TestRouterDelegationTokenSecretManager.java:
##########
@@ -0,0 +1,198 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.secure;
+
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.security.token.delegation.DelegationKey;
+import org.apache.hadoop.test.LambdaTestUtils;
+import org.apache.hadoop.util.Time;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier;
+import org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService;
+import
org.apache.hadoop.yarn.server.router.security.RouterDelegationTokenSecretManager;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+public class TestRouterDelegationTokenSecretManager extends
AbstractSecureRouterTest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(TestRouterDelegationTokenSecretManager.class);
+
+ @Test
+ public void testRouterStoreNewMasterKey() throws Exception {
+ LOG.info("Test RouterDelegationTokenSecretManager: StoreNewMasterKey.");
+
+ // Start the Router in Secure Mode
+ startSecureRouter();
+
+ // Store NewMasterKey
+ RouterClientRMService routerClientRMService =
this.getRouter().getClientRMProxyService();
+ RouterDelegationTokenSecretManager secretManager =
+ routerClientRMService.getRouterDTSecretManager();
+ DelegationKey storeKey = new DelegationKey(1234, 4321,
"keyBytes".getBytes());
+ secretManager.storeNewMasterKey(storeKey);
+
+ // Get DelegationKey
+ DelegationKey paramKey = new DelegationKey(1234, 4321,
"keyBytes".getBytes());
+ DelegationKey responseKey =
secretManager.getMasterKeyByDelegationKey(paramKey);
+
+ Assert.assertNotNull(paramKey);
+ Assert.assertEquals(storeKey.getExpiryDate(), responseKey.getExpiryDate());
+ Assert.assertEquals(storeKey.getKeyId(), responseKey.getKeyId());
+ Assert.assertTrue(Arrays.equals(storeKey.getEncodedKey(),
responseKey.getEncodedKey()));
Review Comment:
Assert.assertArrayEquals
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/TestFederationStateStoreFacade.java:
##########
@@ -232,4 +239,111 @@ public void testGetApplicationHomeSubClusterCache()
throws YarnException {
}
}
+ @Test
+ public void testStoreNewMasterKey() throws YarnException, IOException {
+ // store delegation key;
+ DelegationKey key = new DelegationKey(1234, 4321, "keyBytes".getBytes());
+ HashSet<DelegationKey> keySet = new HashSet<DelegationKey>();
+ keySet.add(key);
+ facade.storeNewMasterKey(key);
+
+ MemoryFederationStateStore federationStateStore =
Review Comment:
Does it need to be MemoryFederationStateStore?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterMasterKeyResponse.java:
##########
@@ -0,0 +1,44 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.federation.store.records;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.yarn.util.Records;
+
[email protected]
[email protected]
+public abstract class RouterMasterKeyResponse {
+
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public static RouterMasterKeyResponse newInstance(RouterMasterKey masterKey)
{
+ RouterMasterKeyResponse request =
Review Comment:
Single line
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterMasterKey.java:
##########
@@ -0,0 +1,128 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.federation.store.records;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.yarn.util.Records;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
[email protected]
[email protected]
+public abstract class RouterMasterKey {
+
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public static RouterMasterKey newInstance(Integer keyId, ByteBuffer
keyBytes, Long expiryDate) {
+ RouterMasterKey policy = Records.newRecord(RouterMasterKey.class);
+ policy.setKeyId(keyId);
+ policy.setKeyBytes(keyBytes);
+ policy.setExpiryDate(expiryDate);
+ return policy;
+ }
+
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public static RouterMasterKey newInstance(RouterMasterKey masterKey) {
+ RouterMasterKey routerMasterKey = Records.newRecord(RouterMasterKey.class);
+ routerMasterKey.setKeyId(masterKey.getKeyId());
+ routerMasterKey.setKeyBytes(masterKey.getKeyBytes());
+ routerMasterKey.setExpiryDate(masterKey.getExpiryDate());
+ return routerMasterKey;
+ }
+
+ /**
+ * Get the keyId of the MasterKey.
+ *
+ * @return MasterKeyId.
+ */
+ @InterfaceAudience.Public
+ @InterfaceStability.Unstable
+ public abstract Integer getKeyId();
+
+ /**
+ * Set the keyId of the MasterKey.
+ *
+ * @param keyId MasterKeyId.
+ */
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public abstract void setKeyId(Integer keyId);
+
+ /**
+ * Get the keyBytes of the DelegationKey.
+ *
+ * @return KeyBytes of the DelegationKey.
+ */
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public abstract ByteBuffer getKeyBytes();
+
+ /**
+ * Set the keyBytes of the DelegationKey.
+ *
+ * @param keyBytes KeyBytes of the DelegationKey.
+ */
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public abstract void setKeyBytes(ByteBuffer keyBytes);
+
+ /**
+ * Get the ExpiryDate of the DelegationKey.
+ *
+ * @return ExpiryDate of the DelegationKey.
+ */
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public abstract Long getExpiryDate();
+
+ /**
+ * Set the expiryDate of the DelegationKey.
+ *
+ * @param expiryDate expiryDate of the DelegationKey.
+ */
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public abstract void setExpiryDate(Long expiryDate);
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = 1;
+ result = prime * result + (int) (getExpiryDate() ^ (getExpiryDate() >>>
32));
+ result = prime * result + Arrays.hashCode(getKeyBytes().array());
+ result = prime * result + getKeyId();
+ return result;
+ }
+
+ @Override
+ public boolean equals(Object right) {
Review Comment:
EqualsBuilder and HashBuilder
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##########
@@ -370,6 +385,142 @@ public GetReservationsHomeSubClusterResponse
getReservationsHomeSubCluster(
return GetReservationsHomeSubClusterResponse.newInstance(result);
}
+ @Override
+ public RouterMasterKeyResponse storeNewMasterKey(RouterMasterKeyRequest
request)
+ throws YarnException, IOException {
+
+ // Restore the DelegationKey from the request
+ RouterMasterKey masterKey = request.getRouterMasterKey();
+ ByteBuffer keyByteBuf = masterKey.getKeyBytes();
Review Comment:
We seem to do this a few times. Can we extract?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/TestFederationStateStoreFacade.java:
##########
@@ -232,4 +239,111 @@ public void testGetApplicationHomeSubClusterCache()
throws YarnException {
}
}
+ @Test
+ public void testStoreNewMasterKey() throws YarnException, IOException {
+ // store delegation key;
+ DelegationKey key = new DelegationKey(1234, 4321, "keyBytes".getBytes());
+ HashSet<DelegationKey> keySet = new HashSet<DelegationKey>();
Review Comment:
new HashSet<>();
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/TestFederationStateStoreFacade.java:
##########
@@ -232,4 +239,111 @@ public void testGetApplicationHomeSubClusterCache()
throws YarnException {
}
}
+ @Test
+ public void testStoreNewMasterKey() throws YarnException, IOException {
+ // store delegation key;
+ DelegationKey key = new DelegationKey(1234, 4321, "keyBytes".getBytes());
+ HashSet<DelegationKey> keySet = new HashSet<DelegationKey>();
+ keySet.add(key);
+ facade.storeNewMasterKey(key);
+
+ MemoryFederationStateStore federationStateStore =
+ (MemoryFederationStateStore) facade.getStateStore();
+ RouterRMDTSecretManagerState secretManagerState =
+ federationStateStore.getRouterRMSecretManagerState();
+ Assert.assertEquals(keySet, secretManagerState.getMasterKeyState());
+ }
+
+ @Test
+ public void testRemoveStoredMasterKey() throws YarnException, IOException {
+ // store delegation key;
+ DelegationKey key = new DelegationKey(4567, 7654, "keyBytes".getBytes());
+ HashSet<DelegationKey> keySet = new HashSet<DelegationKey>();
+ keySet.add(key);
+ facade.storeNewMasterKey(key);
+
+ // check to delete delegationKey
+ facade.removeStoredMasterKey(key);
+ keySet.clear();
+
+ MemoryFederationStateStore federationStateStore =
+ (MemoryFederationStateStore) facade.getStateStore();
+ RouterRMDTSecretManagerState secretManagerState =
+ federationStateStore.getRouterRMSecretManagerState();
+ Assert.assertEquals(keySet, secretManagerState.getMasterKeyState());
+ }
+
+ @Test
+ public void testStoreNewToken() throws YarnException, IOException {
+ // store new rm-token
+ RMDelegationTokenIdentifier dtId1 = new RMDelegationTokenIdentifier(
+ new Text("owner1"), new Text("renewer1"), new Text("realuser1"));
+ int sequenceNumber = 1;
+ dtId1.setSequenceNumber(sequenceNumber);
+ Long renewDate1 = Time.now();
+ facade.storeNewToken(dtId1, renewDate1);
+
+ Map<RMDelegationTokenIdentifier, Long> token1 =
+ new HashMap<RMDelegationTokenIdentifier, Long>();
Review Comment:
new HashMap<>(); and single line
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/secure/TestRouterDelegationTokenSecretManager.java:
##########
@@ -0,0 +1,198 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.secure;
+
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.security.token.delegation.DelegationKey;
+import org.apache.hadoop.test.LambdaTestUtils;
+import org.apache.hadoop.util.Time;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier;
+import org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService;
+import
org.apache.hadoop.yarn.server.router.security.RouterDelegationTokenSecretManager;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+public class TestRouterDelegationTokenSecretManager extends
AbstractSecureRouterTest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(TestRouterDelegationTokenSecretManager.class);
+
+ @Test
+ public void testRouterStoreNewMasterKey() throws Exception {
+ LOG.info("Test RouterDelegationTokenSecretManager: StoreNewMasterKey.");
+
+ // Start the Router in Secure Mode
+ startSecureRouter();
+
+ // Store NewMasterKey
+ RouterClientRMService routerClientRMService =
this.getRouter().getClientRMProxyService();
+ RouterDelegationTokenSecretManager secretManager =
+ routerClientRMService.getRouterDTSecretManager();
+ DelegationKey storeKey = new DelegationKey(1234, 4321,
"keyBytes".getBytes());
+ secretManager.storeNewMasterKey(storeKey);
+
+ // Get DelegationKey
+ DelegationKey paramKey = new DelegationKey(1234, 4321,
"keyBytes".getBytes());
+ DelegationKey responseKey =
secretManager.getMasterKeyByDelegationKey(paramKey);
+
+ Assert.assertNotNull(paramKey);
Review Comment:
Import the asserts statically.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##########
@@ -558,4 +571,32 @@ protected void finalize() {
public Map<String, RequestInterceptorChainWrapper> getUserPipelineMap() {
return userPipelineMap;
}
+
+ /**
+ * Create RouterRMDelegationTokenSecretManager.
+ * In the YARN federation, the Router will replace the RM to
+ * manage the RMDelegationToken (generate, update, cancel),
+ * so the relevant configuration parameters still obtain the configuration
parameters of the RM.
+ *
+ * @param conf Configuration
+ * @return RouterDelegationTokenSecretManager.
+ */
+ protected RouterDelegationTokenSecretManager
createRouterRMDelegationTokenSecretManager(
+ Configuration conf) {
+
+ long secretKeyInterval =
conf.getLong(YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_KEY,
+ YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
+ long tokenMaxLifetime =
conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY,
+ YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
+ long tokenRenewInterval =
conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
+ YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
+
+ return new RouterDelegationTokenSecretManager(secretKeyInterval,
+ tokenMaxLifetime, tokenRenewInterval, 3600000);
Review Comment:
3600000 make it 10 hours: 10 * 60 * 60 * 1000
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/TestFederationStateStoreFacade.java:
##########
@@ -232,4 +239,111 @@ public void testGetApplicationHomeSubClusterCache()
throws YarnException {
}
}
+ @Test
+ public void testStoreNewMasterKey() throws YarnException, IOException {
+ // store delegation key;
+ DelegationKey key = new DelegationKey(1234, 4321, "keyBytes".getBytes());
+ HashSet<DelegationKey> keySet = new HashSet<DelegationKey>();
+ keySet.add(key);
+ facade.storeNewMasterKey(key);
+
+ MemoryFederationStateStore federationStateStore =
+ (MemoryFederationStateStore) facade.getStateStore();
+ RouterRMDTSecretManagerState secretManagerState =
+ federationStateStore.getRouterRMSecretManagerState();
+ Assert.assertEquals(keySet, secretManagerState.getMasterKeyState());
+ }
+
+ @Test
+ public void testRemoveStoredMasterKey() throws YarnException, IOException {
+ // store delegation key;
+ DelegationKey key = new DelegationKey(4567, 7654, "keyBytes".getBytes());
+ HashSet<DelegationKey> keySet = new HashSet<DelegationKey>();
+ keySet.add(key);
+ facade.storeNewMasterKey(key);
+
+ // check to delete delegationKey
+ facade.removeStoredMasterKey(key);
+ keySet.clear();
+
+ MemoryFederationStateStore federationStateStore =
+ (MemoryFederationStateStore) facade.getStateStore();
+ RouterRMDTSecretManagerState secretManagerState =
+ federationStateStore.getRouterRMSecretManagerState();
+ Assert.assertEquals(keySet, secretManagerState.getMasterKeyState());
+ }
+
+ @Test
+ public void testStoreNewToken() throws YarnException, IOException {
+ // store new rm-token
+ RMDelegationTokenIdentifier dtId1 = new RMDelegationTokenIdentifier(
+ new Text("owner1"), new Text("renewer1"), new Text("realuser1"));
+ int sequenceNumber = 1;
+ dtId1.setSequenceNumber(sequenceNumber);
+ Long renewDate1 = Time.now();
+ facade.storeNewToken(dtId1, renewDate1);
+
+ Map<RMDelegationTokenIdentifier, Long> token1 =
+ new HashMap<RMDelegationTokenIdentifier, Long>();
+ token1.put(dtId1, renewDate1);
+
+ MemoryFederationStateStore federationStateStore =
+ (MemoryFederationStateStore) facade.getStateStore();
+ RouterRMDTSecretManagerState storeSecretManagerState =
+ federationStateStore.getRouterRMSecretManagerState();
+ Assert.assertEquals(token1, storeSecretManagerState.getTokenState());
+ }
+
+ @Test
+ public void testUpdateNewToken() throws YarnException, IOException {
+ // store new rm-token
+ RMDelegationTokenIdentifier dtId1 = new RMDelegationTokenIdentifier(
+ new Text("owner2"), new Text("renewer2"), new Text("realuser2"));
+ int sequenceNumber = 2;
+ dtId1.setSequenceNumber(sequenceNumber);
+ Long renewDate1 = Time.now();
+ facade.storeNewToken(dtId1, renewDate1);
+
+ Map<RMDelegationTokenIdentifier, Long> token1 =
+ new HashMap<RMDelegationTokenIdentifier, Long>();
+ token1.put(dtId1, renewDate1);
+
+ renewDate1 = Time.now();
+ facade.updateStoredToken(dtId1, renewDate1);
+ token1.put(dtId1, renewDate1);
+
+ MemoryFederationStateStore federationStateStore =
+ (MemoryFederationStateStore) facade.getStateStore();
+ RouterRMDTSecretManagerState updateSecretManagerState =
+ federationStateStore.getRouterRMSecretManagerState();
+ Assert.assertEquals(token1, updateSecretManagerState.getTokenState());
+ Assert.assertEquals(sequenceNumber,
updateSecretManagerState.getDTSequenceNumber());
+ }
+
+ @Test
+ public void testRemoveStoredToken() throws YarnException, IOException {
+ // store new rm-token
+ RMDelegationTokenIdentifier dtId1 = new RMDelegationTokenIdentifier(
+ new Text("owner3"), new Text("renewer3"), new Text("realuser3"));
+ int sequenceNumber = 3;
+ dtId1.setSequenceNumber(sequenceNumber);
+ Long renewDate1 = Time.now();
+ facade.storeNewToken(dtId1, renewDate1);
+
+ Map<RMDelegationTokenIdentifier, Long> token1 =
+ new HashMap<RMDelegationTokenIdentifier, Long>();
Review Comment:
Single line and shorter.
> Yarn Router Support DelegationToken
> -----------------------------------
>
> Key: YARN-9708
> URL: https://issues.apache.org/jira/browse/YARN-9708
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: router
> Affects Versions: 3.1.1
> Reporter: Xie YiFan
> Assignee: fanshilun
> Priority: Minor
> Labels: pull-request-available
> Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch,
> RMDelegationTokenSecretManager_storeNewMasterKey.svg,
> RouterDelegationTokenSecretManager_storeNewMasterKey.svg
>
>
> 1.we use router as proxy to manage multiple cluster which be independent of
> each other in order to apply unified client. Thus, we implement our
> customized AMRMProxyPolicy that doesn't broadcast ResourceRequest to other
> cluster.
> 2.Our production environment need kerberos. But router doesn't support
> SecureLogin for now.
> https://issues.apache.org/jira/browse/YARN-6539 desn't work. So we
> improvement it.
> 3.Some framework like oozie would get Token via yarnclient#getDelegationToken
> which router doesn't support. Our solution is that adding homeCluster to
> ApplicationSubmissionContextProto & GetDelegationTokenRequestProto. Job would
> be submitted with specified clusterid so that router knows which cluster to
> submit this job. Router would get Token from one RM according to specified
> clusterid when client call getDelegation meanwhile apply some mechanism to
> save this token in memory.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
