[
https://issues.apache.org/jira/browse/HADOOP-18666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700879#comment-17700879
]
ASF GitHub Bot commented on HADOOP-18666:
-----------------------------------------
eubnara commented on PR #5480:
URL: https://github.com/apache/hadoop/pull/5480#issuecomment-1470908526
@tasanuma
I tested on Hadoop 3.3.0 with some patches to fix build failure. But this
issue has been reproduced.
<details><summary>core-site.xml</summary>
```
<configuration xmlns:xi="http://www.w3.org/2001/XInclude";>
<property>
<name>fs.azure.user.agent.prefix</name>
<value>User-Agent: APN/1.0 Hortonworks/1.0 HDP/None</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://ambari-agent-1.example.com:8020</value>
<final>true</final>
</property>
<property>
<name>fs.gs.application.name.suffix</name>
<value> (GPN:Hortonworks; version 1.0) HDP/None</value>
</property>
<property>
<name>fs.gs.path.encoding</name>
<value>uri-path</value>
</property>
<property>
<name>fs.gs.working.dir</name>
<value>/</value>
</property>
<property>
<name>fs.s3a.user.agent.prefix</name>
<value>User-Agent: APN/1.0 Hortonworks/1.0 HDP/None</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>360</value>
</property>
<property>
<name>ha.failover-controller.active-standby-elector.zk.op.retries</name>
<value>120</value>
</property>
<property>
<name>hadoop.http.authentication.kerberos.endpoint.whitelist</name>
<value>/isActive,/jmx,/prom</value>
</property>
<property>
<name>hadoop.http.authentication.kerberos.keytab</name>
<value>/etc/security/keytabs/spnego.service.keytab</value>
</property>
<property>
<name>hadoop.http.authentication.kerberos.principal</name>
<value>HTTP/[email protected]</value>
</property>
<property>
<name>hadoop.http.authentication.type</name>
<value>kerberos</value>
</property>
<property>
<name>hadoop.http.filter.initializers</name>
<value>org.apache.hadoop.security.AuthenticationFilterInitializer,org.apache.hadoop.security.HttpCrossOriginFilterInitializer</value>
</property>
<property>
<name>hadoop.proxyuser.*</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hdfs.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hdfs.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.HTTP.groups</name>
<value>hadoop</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>ambari-server.example.com</value>
</property>
<property>
<name>hadoop.proxyuser.yarn.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.yarn.hosts</name>
<value>ambari-agent-2.example.com</value>
</property>
<property>
<name>hadoop.security.auth_to_local</name>
<value>RULE:[1:$1@$0]([email protected])s/.*/ambari-qa/
RULE:[1:$1@$0]([email protected])s/.*/hdfs/
RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*//
RULE:[2:$1@$0]([email protected])s/.*/hdfs/
RULE:[2:$1@$0]([email protected])s/.*/mapred/
RULE:[2:$1@$0]([email protected])s/.*/yarn/
RULE:[2:$1@$0]([email protected])s/.*/hdfs/
RULE:[2:$1@$0]([email protected])s/.*/yarn/
DEFAULT</value>
</property>
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hadoop.security.authorization</name>
<value>true</value>
</property>
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>io.serializations</name>
<value>org.apache.hadoop.io.serializer.WritableSerialization</value>
</property>
<property>
<name>ipc.client.connect.max.retries</name>
<value>50</value>
</property>
<property>
<name>ipc.client.connection.maxidletime</name>
<value>30000</value>
</property>
<property>
<name>ipc.client.idlethreshold</name>
<value>8000</value>
</property>
<property>
<name>ipc.server.tcpnodelay</name>
<value>true</value>
</property>
<property>
<name>mapreduce.jobtracker.webinterface.trusted</name>
<value>false</value>
</property>
<property>
<name>net.topology.script.file.name</name>
<value>/etc/hadoop/conf/topology_script.py</value>
</property>
</configuration>
```
</details>
<details><summary>yarn-site.xml</summary>
```
<configuration xmlns:xi="http://www.w3.org/2001/XInclude";>
<property>
<name>hadoop.registry.secure</name>
<value>true</value>
</property>
<property>
<name>hadoop.registry.system.accounts</name>
<value>sasl:yarn,sasl:mapred,sasl:hadoop,sasl:hdfs,sasl:rm</value>
</property>
<property>
<name>manage.include.files</name>
<value>false</value>
</property>
<property>
<name>yarn.acl.enable</name>
<value>true</value>
</property>
<property>
<name>yarn.admin.acl</name>
<value></value>
</property>
<property>
<name>yarn.application.classpath</name>
<value>/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*,/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*</value>
</property>
<property>
<name>yarn.http.policy</name>
<value>HTTP_ONLY</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>2592000</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://ambari-agent-2.example.com:19888/jobhistory/logs</value>
</property>
<property>
<name>yarn.nodemanager.address</name>
<value>0.0.0.0:45454</value>
</property>
<property>
<name>yarn.nodemanager.admin-env</name>
<value>MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.container-executor.class</name>
<value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
</property>
<property>
<name>yarn.nodemanager.container-monitor.interval-ms</name>
<value>3000</value>
</property>
<property>
<name>yarn.nodemanager.delete.debug-delay-sec</name>
<value>0</value>
</property>
<property>
<name>yarn.nodemanager.disk-health-checker.min-healthy-disks</name>
<value>0.25</value>
</property>
<property>
<name>yarn.nodemanager.health-checker.interval-ms</name>
<value>135000</value>
</property>
<property>
<name>yarn.nodemanager.health-checker.script.timeout-ms</name>
<value>60000</value>
</property>
<property>
<name>yarn.nodemanager.keytab</name>
<value>/etc/security/keytabs/nm.service.keytab</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.group</name>
<value>hadoop</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/hadoop/yarn/local</value>
</property>
<property>
<name>yarn.nodemanager.log-aggregation.compression-type</name>
<value>gz</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/hadoop/yarn/log</value>
</property>
<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>604800</value>
</property>
<property>
<name>yarn.nodemanager.principal</name>
<value>nm/[email protected]</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/app-logs</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir-suffix</name>
<value>logs</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>12288</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>2.1</value>
</property>
<property>
<name>yarn.nodemanager.webapp.spnego-keytab-file</name>
<value>/etc/security/keytabs/spnego.service.keytab</value>
</property>
<property>
<name>yarn.nodemanager.webapp.spnego-principal</name>
<value>HTTP/[email protected]</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>ambari-agent-2.example.com:8050</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>ambari-agent-2.example.com:8141</value>
</property>
<property>
<name>yarn.resourcemanager.am.max-attempts</name>
<value>2</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>ambari-agent-2.example.com</value>
</property>
<property>
<name>yarn.resourcemanager.keytab</name>
<value>/etc/security/keytabs/rm.service.keytab</value>
</property>
<property>
<name>yarn.resourcemanager.nodes.exclude-path</name>
<value>/etc/hadoop/conf/yarn.exclude</value>
</property>
<property>
<name>yarn.resourcemanager.principal</name>
<value>rm/[email protected]</value>
</property>
<property>
<name>yarn.resourcemanager.proxy-user-privileges.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.proxyuser.*.groups</name>
<value></value>
</property>
<property>
<name>yarn.resourcemanager.proxyuser.*.hosts</name>
<value></value>
</property>
<property>
<name>yarn.resourcemanager.proxyuser.*.users</name>
<value></value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>ambari-agent-2.example.com:8025</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>ambari-agent-2.example.com:8030</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>ambari-agent-2.example.com:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>ambari-agent-2.example.com:8090</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.spnego-keytab-file</name>
<value>/etc/security/keytabs/spnego.service.keytab</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.spnego-principal</name>
<value>HTTP/[email protected]</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>12288</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>4096</value>
</property>
<property>
<name>yarn.timeline-service.address</name>
<value>localhost:10200</value>
</property>
<property>
<name>yarn.timeline-service.enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.timeline-service.entity-group-fs-store.active-dir</name>
<value>/ats/active/</value>
</property>
<property>
<name>yarn.timeline-service.entity-group-fs-store.cleaner-interval-seconds</name>
<value>3600</value>
</property>
<property>
<name>yarn.timeline-service.entity-group-fs-store.done-dir</name>
<value>/ats/done/</value>
</property>
<property>
<name>yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes</name>
<value></value>
</property>
<property>
<name>yarn.timeline-service.entity-group-fs-store.retain-seconds</name>
<value>604800</value>
</property>
<property>
<name>yarn.timeline-service.entity-group-fs-store.scan-interval-seconds</name>
<value>60</value>
</property>
<property>
<name>yarn.timeline-service.entity-group-fs-store.summary-store</name>
<value>org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore</value>
</property>
<property>
<name>yarn.timeline-service.generic-application-history.store-class</name>
<value>org.apache.hadoop.yarn.server.applicationhistoryservice.NullApplicationHistoryStore</value>
</property>
<property>
<name>yarn.timeline-service.http-authentication.cookie.domain</name>
<value></value>
</property>
<property>
<name>yarn.timeline-service.http-authentication.cookie.path</name>
<value></value>
</property>
<property>
<name>yarn.timeline-service.http-authentication.kerberos.name.rules</name>
<value></value>
</property>
<property>
<name>yarn.timeline-service.http-authentication.proxyuser.*.groups</name>
<value></value>
</property>
<property>
<name>yarn.timeline-service.http-authentication.proxyuser.*.hosts</name>
<value></value>
</property>
<property>
<name>yarn.timeline-service.http-authentication.proxyuser.*.users</name>
<value></value>
</property>
<property>
<name>yarn.timeline-service.http-authentication.signature.secret</name>
<value></value>
</property>
<property>
<name>yarn.timeline-service.http-authentication.signature.secret.file</name>
<value></value>
</property>
<property>
<name>yarn.timeline-service.http-authentication.signer.secret.provider</name>
<value></value>
</property>
<property>
<name>yarn.timeline-service.http-authentication.signer.secret.provider.object</name>
<value></value>
</property>
<property>
<name>yarn.timeline-service.http-authentication.token.validity</name>
<value></value>
</property>
<property>
<name>yarn.timeline-service.http-authentication.type</name>
<value>kerberos</value>
</property>
<property>
<name>yarn.timeline-service.leveldb-timeline-store.path</name>
<value>/var/log/hadoop-yarn/timeline</value>
</property>
<property>
<name>yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms</name>
<value>300000</value>
</property>
<property>
<name>yarn.timeline-service.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.timeline-service.store-class</name>
<value>org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore</value>
</property>
<property>
<name>yarn.timeline-service.ttl-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.timeline-service.ttl-ms</name>
<value>2678400000</value>
</property>
<property>
<name>yarn.timeline-service.webapp.address</name>
<value>localhost:8188</value>
</property>
<property>
<name>yarn.timeline-service.webapp.https.address</name>
<value>localhost:8190</value>
</property>
</configuration>
```
</details>
The source code I use when testing it is here =>
https://github.com/eubnara/hadoop/tree/eub-3.3.0.
I build it with apache/bigtop project.
> A whitelist of endpoints to skip Kerberos authentication doesn't work for
> ResourceManager and Job History Server
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-18666
> URL: https://issues.apache.org/jira/browse/HADOOP-18666
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
> Reporter: YUBI LEE
> Assignee: YUBI LEE
> Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-18666-branch-3.3.4.patch
>
>
> Thanks to HADOOP-16527, we can add a whitelist of endpoints to skip Kerberos
> authentication such as {{/isActive}}, {{/jmx}}, {{/prom}}.
> However, I found that ResourceManager and Job History Server doesn't repect
> {{hadoop.http.authentication.kerberos.endpoint.whitelist}}.
> To workaround this issue for ResourceManager, set
> {{yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled=true}} in
> yarn-site.xml.
> However, there is no workaround for Job History Server.
> This bug is caused by {{HttpServer2#initSpnego}} call without proper
> configurations which starts with "{{hadoop.http.authentication.}}".
> I will make a PR soon.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
