[
https://issues.apache.org/jira/browse/HADOOP-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18009531#comment-18009531
]
László Bodor commented on HADOOP-19639:
---------------------------------------
let me add the motivation from Tez side as a context, so as [~bkosztolnik]
mentioned we faced a problem after YARN-11738,
because Tez is supposed to work by runtime payloads (containing all the hadoop
config), but in the hadoop layer, the options are initialized in the static
initializer, so tez cannot help with that exception, because new
Configuration() depends on the core-site.xml config which is *maybe* on the
classpath, otherwise, everything turns to default
first I tried to make the TezClientToAMTokenSecretManager extend a custom Tez
implementation of SecretManager, but it led to compilation issues, as Hadoop
layers expect a Hadoop SecretManager (see RPC), so eventually we cannot pass a
Tez SecretManager that doesn’t inherit Hadoop’s SecretManager, but as long as
we extend the Hadoop one, the static initializer and then the field initializer
of keyGen keyGen.init(SELECTED_LENGTH) will kick in immediately
so this cannot be fixed from Tez, we have 2 options:
1. rework YARN-11738 (as this problem was also implied in this comment upstream)
2. make the core-site.xml localized to tez containers to have it picked up <-
this is against tez design, so I would prefer 1)
optimal way would be completely eliminated statics from SecretManager, but I'm
afraid they are there for a reason, so basically, anything could work for us
which makes Tez able to intercept, and configure the SecretManager from a
Configuration object, which is different than the default one (which is
instantiated by new Configuration())
so this cannot be fixed from Tez, we have 2 options:
1. rework YARN-11738 (as this problem was also implied in this comment upstream)
2. make the core-site.xml localized to tez containers to have it picked up
I’m a bit against 2), because it’s also a design decision to make file config
resources available to tez containers instead of payload, so I would definitely
be in favor of 1), here is where I need the opinion of Hadoop folks
> SecretManager configuration at runtime
> --------------------------------------
>
> Key: HADOOP-19639
> URL: https://issues.apache.org/jira/browse/HADOOP-19639
> Project: Hadoop Common
> Issue Type: Improvement
> Components: hadoop-common
> Affects Versions: 3.5.0
> Reporter: Bence Kosztolnik
> Assignee: Bence Kosztolnik
> Priority: Major
> Labels: pull-request-available
>
> In case of TEZ *DAGAppMaster* the Hadoop *SecretManager* code can not read
> yarn config xml file, therefore the SELECTED_ALGORITHM and SELECTED_LENGTH
> variables in SecretManager can not be set at runtime.
> This can results with the following exception in FIPS environment:
> {code:java}
> java.security.InvalidParameterException: Key size for HMAC must be at least
> 112 bits in approved mode: SHA-1/HMAC
> at
> com.safelogic.cryptocomply.fips.core/com.safelogic.cryptocomply.jcajce.provider.BaseKeyGenerator.engineInit(Unknown
> Source)
> at java.base/javax.crypto.KeyGenerator.init(KeyGenerator.java:540)
> at java.base/javax.crypto.KeyGenerator.init(KeyGenerator.java:517)
> at
> org.apache.hadoop.security.token.SecretManager.<init>(SecretManager.java:157)
> at
> org.apache.hadoop.yarn.security.client.BaseClientToAMTokenSecretManager.<init>(BaseClientToAMTokenSecretManager.java:38)
> at
> org.apache.hadoop.yarn.security.client.ClientToAMTokenSecretManager.<init>(ClientToAMTokenSecretManager.java:46)
> at
> org.apache.tez.common.security.TezClientToAMTokenSecretManager.<init>(TezClientToAMTokenSecretManager.java:33)
> at
> org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:493)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2649)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1910)
> at
> org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2646)
> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2440)
> {code}
> To mitigate the problem we should modify the *ClientToAMTokenSecretManager*
> to have a constructor where TEZ can path a configuration object with the
> selected values.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
