[jira] [Updated] (YARN-11155) ATS v1.5 doesn't work with JWTRedirectAuthenticationHandler
[
https://issues.apache.org/jira/browse/YARN-11155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated YARN-11155:
---
Attachment: YARN-11155.001.patch
> ATS v1.5 doesn't work with JWTRedirectAuthenticationHandler
> ---
>
> Key: YARN-11155
> URL: https://issues.apache.org/jira/browse/YARN-11155
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2, 3.3.2
>Reporter: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-11155.001.patch
>
>
> When ATS is configured with JWTRedirectAuthenticationHandler for KnoxSSO, In
> ATS, Delegation Token operation does not work.
> In this situation, All hadoop web daemon use JWTRedirectAuthenticationHandler
> for KnoxSSO. But ATS should be use kerberos auth handler. Tez job users
> should login to kerberos for spnego auth for tez-ui access in own local pc.
> It is very inconvenient.
>
> Expected result (use JWTRedirectAuthenticationHandler)
> {code:java}
> curl -s -u: --negotiate
> "https://ats.host.com:8190/ws/v1/timeline/?op=GETDELEGATIONTOKEN&=rm%2Frm1.host.com%40EXAMPLE.ORG;
> {
> "Token": {
> "urlString": "KAbnVtLWFkbWm8EsIAZVElNfREVMRUTl9UT0tFTgA"
> }
> }
> {code}
>
> Wrong result (use JWTRedirectAuthenticationHandler)
> {code:java}
> curl -s -u: --negotiate
> "https://ats.host.com:8190/ws/v1/timeline/?op=GETDELEGATIONTOKEN&=rm%2Frm1.host.com%40EXAMPLE.ORG;
> {
> "About": "Timeline API",
> "hadoop-build-version": "3.1.2 from
> 7c62584effd9a5aa4b90d22dbf8d8eb2bca03feb by irteam source checksum
> 444e3aaa7feb4f8f73c3c3a71dbdd38",
> "hadoop-version": "3.1.2",
> "hadoop-version-built-on": "2022-04-08T03:45Z",
> "timeline-service-build-version": "3.1.2-49 from
> 7c62584effd9a5aa4b90d22dbf8d8eb2bca03feb by users source checksum
> 7594ee7186b86eeccfc787d139ee8b",
> "timeline-service-version": "3.1.2",
> "timeline-service-version-built-on": "2022-04-08T03:49Z"
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11155) ATS v1.5 doesn't work with JWTRedirectAuthenticationHandler
KWON BYUNGCHANG created YARN-11155:
--
Summary: ATS v1.5 doesn't work with
JWTRedirectAuthenticationHandler
Key: YARN-11155
URL: https://issues.apache.org/jira/browse/YARN-11155
Project: Hadoop YARN
Issue Type: Bug
Components: timelineserver
Affects Versions: 3.3.2, 3.1.2
Reporter: KWON BYUNGCHANG
When ATS is configured with JWTRedirectAuthenticationHandler for KnoxSSO, In
ATS, Delegation Token operation does not work.
In this situation, All hadoop web daemon use JWTRedirectAuthenticationHandler
for KnoxSSO. But ATS should be use kerberos auth handler. Tez job users should
login to kerberos for spnego auth for tez-ui access in own local pc. It is very
inconvenient.
Expected result (use JWTRedirectAuthenticationHandler)
{code:java}
curl -s -u: --negotiate
"https://ats.host.com:8190/ws/v1/timeline/?op=GETDELEGATIONTOKEN&=rm%2Frm1.host.com%40EXAMPLE.ORG;
{
"Token": {
"urlString": "KAbnVtLWFkbWm8EsIAZVElNfREVMRUTl9UT0tFTgA"
}
}
{code}
Wrong result (use JWTRedirectAuthenticationHandler)
{code:java}
curl -s -u: --negotiate
"https://ats.host.com:8190/ws/v1/timeline/?op=GETDELEGATIONTOKEN&=rm%2Frm1.host.com%40EXAMPLE.ORG;
{
"About": "Timeline API",
"hadoop-build-version": "3.1.2 from
7c62584effd9a5aa4b90d22dbf8d8eb2bca03feb by irteam source checksum
444e3aaa7feb4f8f73c3c3a71dbdd38",
"hadoop-version": "3.1.2",
"hadoop-version-built-on": "2022-04-08T03:45Z",
"timeline-service-build-version": "3.1.2-49 from
7c62584effd9a5aa4b90d22dbf8d8eb2bca03feb by users source checksum
7594ee7186b86eeccfc787d139ee8b",
"timeline-service-version": "3.1.2",
"timeline-service-version-built-on": "2022-04-08T03:49Z"
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10899) control whether non-exclusive allocation is used
[ https://issues.apache.org/jira/browse/YARN-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-10899: --- Attachment: YARN-10899.004.patch > control whether non-exclusive allocation is used > > > Key: YARN-10899 > URL: https://issues.apache.org/jira/browse/YARN-10899 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: KWON BYUNGCHANG >Priority: Major > Attachments: YARN-10899.001.patch, YARN-10899.002.patch, > YARN-10899.003.patch, YARN-10899.004.patch > > > A non-exclusive partition has the advantage of increasing resource > utilization. > But, it is not useful in all use cases. > In the case of a long-running container, > if it is allocated with non-exclusive allocation, it is more likely to be > preempted and the overhead of repeating the same operation occurs. > A function that can control whether non-exclusive allocation is used from the > user's point of view is required. > I suggest: > When node label expression is empty string, only default partition without > non-exclusive allocation is used, > when node label expression is null, only it allowed non-exclusive allocation. > I will attach patch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10899) control whether non-exclusive allocation is used
[ https://issues.apache.org/jira/browse/YARN-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-10899: --- Attachment: YARN-10899.003.patch > control whether non-exclusive allocation is used > > > Key: YARN-10899 > URL: https://issues.apache.org/jira/browse/YARN-10899 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: KWON BYUNGCHANG >Priority: Major > Attachments: YARN-10899.001.patch, YARN-10899.002.patch, > YARN-10899.003.patch > > > A non-exclusive partition has the advantage of increasing resource > utilization. > But, it is not useful in all use cases. > In the case of a long-running container, > if it is allocated with non-exclusive allocation, it is more likely to be > preempted and the overhead of repeating the same operation occurs. > A function that can control whether non-exclusive allocation is used from the > user's point of view is required. > I suggest: > When node label expression is empty string, only default partition without > non-exclusive allocation is used, > when node label expression is null, only it allowed non-exclusive allocation. > I will attach patch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10898) Support standalone YARN Service API SERVER
[ https://issues.apache.org/jira/browse/YARN-10898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-10898: --- Description: Currently, YARN Service API server is served by ResourceManager. If API Server can be operated independently of Resource Manager. there are serveral advantages. Bootstrap code for running YARN Service APIServer standalone is already prepared, but it is not exposed as yarn cli subcommand. Added apiserver subcommand to yarn cli command and fixed bugs in bootstrap code. 1. APIServer does not run as a daemon, jvm process is terminated 2. Added SSL certificate configuration 3. Added authentication handler was: Bootstrap code for running YARN Service APIServer standalone is alreadt prepared, but it is not exposed as yarn cli subcommand. Added apiserver subcommand to yarn cli command and fixed bugs in bootstrap code. 1. APIServer does not run as a daemon, jvm process is terminated 2. Added SSL certificate configuration 3. Added authentication handler > Support standalone YARN Service API SERVER > -- > > Key: YARN-10898 > URL: https://issues.apache.org/jira/browse/YARN-10898 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-native-services >Reporter: KWON BYUNGCHANG >Priority: Major > Attachments: YARN-10898.001.patch > > > Currently, YARN Service API server is served by ResourceManager. If API > Server can be operated independently of Resource Manager. there are serveral > advantages. > Bootstrap code for running YARN Service APIServer standalone is already > prepared, but it is not exposed as yarn cli subcommand. > Added apiserver subcommand to yarn cli command and fixed bugs in bootstrap > code. > 1. APIServer does not run as a daemon, jvm process is terminated > 2. Added SSL certificate configuration > 3. Added authentication handler -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10899) control whether non-exclusive allocation is used
[ https://issues.apache.org/jira/browse/YARN-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-10899: --- Description: A non-exclusive partition has the advantage of increasing resource utilization. But, it is not useful in all use cases. In the case of a long-running container, if it is allocated with non-exclusive allocation, it is more likely to be preempted and the overhead of repeating the same operation occurs. A function that can control whether non-exclusive allocation is used from the user's point of view is required. I suggest: When node label expression is empty string, only default partition without non-exclusive allocation is used, when node label expression is null, only it allowed non-exclusive allocation. I will attach patch. was: A non-exclusive partition has the advantage of increasing resource utilization. But, it is not useful in all use cases. In the case of a long-running container, if it is allocated with non-exclusive allocation, it is more likely to be preempted and the overhead of repeating the same operation occurs. A function that can control whether non-exclusive allocation is used from the user's point of view is required. I suggest: When node label expression is empty string, only default partition without non-exclusive partition is used, when node label expression is null, only it allowed non-exclusive allocation. I will attach patch. > control whether non-exclusive allocation is used > > > Key: YARN-10899 > URL: https://issues.apache.org/jira/browse/YARN-10899 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: KWON BYUNGCHANG >Priority: Major > Attachments: YARN-10899.001.patch, YARN-10899.002.patch > > > A non-exclusive partition has the advantage of increasing resource > utilization. > But, it is not useful in all use cases. > In the case of a long-running container, > if it is allocated with non-exclusive allocation, it is more likely to be > preempted and the overhead of repeating the same operation occurs. > A function that can control whether non-exclusive allocation is used from the > user's point of view is required. > I suggest: > When node label expression is empty string, only default partition without > non-exclusive allocation is used, > when node label expression is null, only it allowed non-exclusive allocation. > I will attach patch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10899) control whether non-exclusive allocation is used
[ https://issues.apache.org/jira/browse/YARN-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-10899: --- Attachment: YARN-10899.002.patch > control whether non-exclusive allocation is used > > > Key: YARN-10899 > URL: https://issues.apache.org/jira/browse/YARN-10899 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: KWON BYUNGCHANG >Priority: Major > Attachments: YARN-10899.001.patch, YARN-10899.002.patch > > > A non-exclusive partition has the advantage of increasing resource > utilization. > But, it is not useful in all use cases. > In the case of a long-running container, > if it is allocated with non-exclusive allocation, it is more likely to be > preempted and the overhead of repeating the same operation occurs. > A function that can control whether non-exclusive allocation is used from the > user's point of view is required. > I suggest: > When node label expression is empty string, only default partition without > non-exclusive partition is used, > when node label expression is null, only it allowed non-exclusive allocation. > I will attach patch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10899) control whether non-exclusive allocation is used
[ https://issues.apache.org/jira/browse/YARN-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-10899: --- Summary: control whether non-exclusive allocation is used (was: controll whether non-exclusive allocation is used) > control whether non-exclusive allocation is used > > > Key: YARN-10899 > URL: https://issues.apache.org/jira/browse/YARN-10899 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: KWON BYUNGCHANG >Priority: Major > > A non-exclusive partition has the advantage of increasing resource > utilization. > But, it is not useful in all use cases. > In the case of a long-running container, > if it is allocated with non-exclusive allocation, it is more likely to be > preempted and the overhead of repeating the same operation occurs. > A function that can control whether non-exclusive allocation is used from the > user's point of view is required. > I suggest: > When node label expression is empty string, only default partition without > non-exclusive partition is used, > when node label expression is null, only it allowed non-exclusive allocation. > I will attach patch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10899) controll whether non-exclusive allocation is used
KWON BYUNGCHANG created YARN-10899: -- Summary: controll whether non-exclusive allocation is used Key: YARN-10899 URL: https://issues.apache.org/jira/browse/YARN-10899 Project: Hadoop YARN Issue Type: Improvement Reporter: KWON BYUNGCHANG A non-exclusive partition has the advantage of increasing resource utilization. But, it is not useful in all use cases. In the case of a long-running container, if it is allocated with non-exclusive allocation, it is more likely to be preempted and the overhead of repeating the same operation occurs. A function that can control whether non-exclusive allocation is used from the user's point of view is required. I suggest: When node label expression is empty string, only default partition without non-exclusive partition is used, when node label expression is null, only it allowed non-exclusive allocation. I will attach patch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10898) Support standalone YARN Service API SERVER
KWON BYUNGCHANG created YARN-10898: -- Summary: Support standalone YARN Service API SERVER Key: YARN-10898 URL: https://issues.apache.org/jira/browse/YARN-10898 Project: Hadoop YARN Issue Type: Improvement Components: yarn-native-services Reporter: KWON BYUNGCHANG Bootstrap code for running YARN Service APIServer standalone is alreadt prepared, but it is not exposed as yarn cli subcommand. Added apiserver subcommand to yarn cli command and fixed bugs in bootstrap code. 1. APIServer does not run as a daemon, jvm process is terminated 2. Added SSL certificate configuration 3. Added authentication handler -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10410) Does not work specifying the docker client configuration in distributed shell
[
https://issues.apache.org/jira/browse/YARN-10410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated YARN-10410:
---
Attachment: YARN-10410.002.patch
> Does not work specifying the docker client configuration in distributed shell
> -
>
> Key: YARN-10410
> URL: https://issues.apache.org/jira/browse/YARN-10410
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-10410.001.patch, YARN-10410.002.patch
>
>
> Job sumbit client side log
> {noformat}
> 2020-08-26 12:07:30,756 INFO distributedshell.Client: Got dt for hdfs://at;
> Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:at, Ident: (token for magnum:
> HDFS_DELEGATION_TOKEN owner=mag...@test.com, renewer=yarn, realUser=,
> issueDate=1598411249950, maxDate=1599275249950, sequenceNumber=1359401,
> masterKeyId=461)
> 2020-08-26 12:07:30,757 INFO distributedshell.Client: Got dt for hdfs://at;
> Kind: kms-dt, Service: kms://ht...@ranger-kms.test.com:9393/kms, Ident:
> (kms-dt owner=magnum, renewer=yarn, realUser=, issueDate=1598411250569,
> maxDate=1599016050569, sequenceNumber=2791602, masterKeyId=908)
> 2020-08-26 12:07:30,969 INFO util.DockerClientConfigHandler: Token read from
> Docker client configuration file: Kind: DOCKER_CLIENT_CREDENTIAL_TOKEN,
> Service: reg.test.com, Ident: (registryUrl: "reg.test.com" applicationId:
> "application_1593772175687_475926")
> {noformat}
>
> AM LOG
> {noformat}
> 20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Executing with
> tokens:
> 20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind:
> YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id:
> 475926 cluster_timestamp: 1593772175687 } attemptId: 1 } keyId: 1589057158)
> 20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind: kms-dt,
> Service: kms://ht...@ranger-kms.test.com:9393/kms, Ident: (kms-dt
> owner=magnum, renewer=yarn, realUser=, issueDate=1598411250569,
> maxDate=1599016050569, sequenceNumber=2791602, masterKeyId=908)
> 20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:at, Ident: (token for magnum:
> HDFS_DELEGATION_TOKEN owner=mag...@test.com, renewer=yarn, realUser=,
> issueDate=1598411249950, maxDate=1599275249950, sequenceNumber=1359401,
> masterKeyId=461)
> 20/08/26 12:07:33 INFO impl.NMClientAsyncImpl: Upper bound of the thread pool
> size is 500
> 20/08/26 12:07:33 WARN distributedshell.ApplicationMaster: Timeline service
> is not enabled
> {noformat}
>
> On the client side in distributea shell, Client read docker_client_config and
> put it in credentials instances
> DOCKER_CLIENT_CREDENTIAL_TOKEN is not propagated to the container.
> eventually docker container launch that requires auth token failed.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10410) Does not work specifying the docker client configuration in distributed shell
[
https://issues.apache.org/jira/browse/YARN-10410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated YARN-10410:
---
Description:
Job sumbit client side log
{noformat}
2020-08-26 12:07:30,756 INFO distributedshell.Client: Got dt for hdfs://at;
Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:at, Ident: (token for magnum:
HDFS_DELEGATION_TOKEN owner=mag...@test.com, renewer=yarn, realUser=,
issueDate=1598411249950, maxDate=1599275249950, sequenceNumber=1359401,
masterKeyId=461)
2020-08-26 12:07:30,757 INFO distributedshell.Client: Got dt for hdfs://at;
Kind: kms-dt, Service: kms://ht...@ranger-kms.test.com:9393/kms, Ident: (kms-dt
owner=magnum, renewer=yarn, realUser=, issueDate=1598411250569,
maxDate=1599016050569, sequenceNumber=2791602, masterKeyId=908)
2020-08-26 12:07:30,969 INFO util.DockerClientConfigHandler: Token read from
Docker client configuration file: Kind: DOCKER_CLIENT_CREDENTIAL_TOKEN,
Service: reg.test.com, Ident: (registryUrl: "reg.test.com" applicationId:
"application_1593772175687_475926")
{noformat}
AM LOG
{noformat}
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Executing with
tokens:
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind:
YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 475926
cluster_timestamp: 1593772175687 } attemptId: 1 } keyId: 1589057158)
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind: kms-dt,
Service: kms://ht...@ranger-kms.test.com:9393/kms, Ident: (kms-dt owner=magnum,
renewer=yarn, realUser=, issueDate=1598411250569, maxDate=1599016050569,
sequenceNumber=2791602, masterKeyId=908)
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind:
HDFS_DELEGATION_TOKEN, Service: ha-hdfs:at, Ident: (token for magnum:
HDFS_DELEGATION_TOKEN owner=mag...@test.com, renewer=yarn, realUser=,
issueDate=1598411249950, maxDate=1599275249950, sequenceNumber=1359401,
masterKeyId=461)
20/08/26 12:07:33 INFO impl.NMClientAsyncImpl: Upper bound of the thread pool
size is 500
20/08/26 12:07:33 WARN distributedshell.ApplicationMaster: Timeline service is
not enabled
{noformat}
On the client side in distributea shell, Client read docker_client_config and
put it in credentials instances
DOCKER_CLIENT_CREDENTIAL_TOKEN is not propagated to the container.
eventually docker container launch that requires auth token failed.
was:
Job sumbit client side log
{noformat}
2020-08-26 12:07:30,756 INFO distributedshell.Client: Got dt for hdfs://at;
Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:at, Ident: (token for magnum:
HDFS_DELEGATION_TOKEN owner=mag...@test.com, renewer=yarn, realUser=,
issueDate=1598411249950, maxDate=1599275249950, sequenceNumber=1359401,
masterKeyId=461)
2020-08-26 12:07:30,757 INFO distributedshell.Client: Got dt for hdfs://at;
Kind: kms-dt, Service: kms://ht...@ranger-kms.test.com:9393/kms, Ident: (kms-dt
owner=magnum, renewer=yarn, realUser=, issueDate=1598411250569,
maxDate=1599016050569, sequenceNumber=2791602, masterKeyId=908)
2020-08-26 12:07:30,969 INFO util.DockerClientConfigHandler: Token read from
Docker client configuration file: Kind: DOCKER_CLIENT_CREDENTIAL_TOKEN,
Service: reg.test.com, Ident: (registryUrl: "reg.test.com" applicationId:
"application_1593772175687_475926")
{noformat}
AM LOG
{noformat}
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Executing with
tokens:
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind:
YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 475926
cluster_timestamp: 1593772175687 } attemptId: 1 } keyId: 1589057158)
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind: kms-dt,
Service: kms://ht...@ranger-kms.test.com:9393/kms, Ident: (kms-dt owner=magnum,
renewer=yarn, realUser=, issueDate=1598411250569, maxDate=1599016050569,
sequenceNumber=2791602, masterKeyId=908)
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind:
HDFS_DELEGATION_TOKEN, Service: ha-hdfs:at, Ident: (token for magnum:
HDFS_DELEGATION_TOKEN owner=mag...@test.com, renewer=yarn, realUser=,
issueDate=1598411249950, maxDate=1599275249950, sequenceNumber=1359401,
masterKeyId=461)
20/08/26 12:07:33 INFO impl.NMClientAsyncImpl: Upper bound of the thread pool
size is 500
20/08/26 12:07:33 WARN distributedshell.ApplicationMaster: Timeline service is
not enabled
{noformat}
DOCKER_CLIENT_CREDENTIAL_TOKEN is not propagated to the container.
eventually docker container launch that requires auth token failed.
> Does not work specifying the docker client configuration in distributed shell
> -
>
> Key: YARN-10410
> URL: https://issues.apache.org/jira/browse/YARN-10410
> Project: Hadoop YARN
> Issue Type: Bug
> Components:
[jira] [Updated] (YARN-10410) Does not work specifying the docker client configuration in distributed shell
[
https://issues.apache.org/jira/browse/YARN-10410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated YARN-10410:
---
Description:
Job sumbit client side log
{noformat}
2020-08-26 12:07:30,756 INFO distributedshell.Client: Got dt for hdfs://at;
Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:at, Ident: (token for magnum:
HDFS_DELEGATION_TOKEN owner=mag...@test.com, renewer=yarn, realUser=,
issueDate=1598411249950, maxDate=1599275249950, sequenceNumber=1359401,
masterKeyId=461)
2020-08-26 12:07:30,757 INFO distributedshell.Client: Got dt for hdfs://at;
Kind: kms-dt, Service: kms://ht...@ranger-kms.test.com:9393/kms, Ident: (kms-dt
owner=magnum, renewer=yarn, realUser=, issueDate=1598411250569,
maxDate=1599016050569, sequenceNumber=2791602, masterKeyId=908)
2020-08-26 12:07:30,969 INFO util.DockerClientConfigHandler: Token read from
Docker client configuration file: Kind: DOCKER_CLIENT_CREDENTIAL_TOKEN,
Service: reg.test.com, Ident: (registryUrl: "reg.test.com" applicationId:
"application_1593772175687_475926")
{noformat}
AM LOG
{noformat}
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Executing with
tokens:
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind:
YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 475926
cluster_timestamp: 1593772175687 } attemptId: 1 } keyId: 1589057158)
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind: kms-dt,
Service: kms://ht...@ranger-kms.test.com:9393/kms, Ident: (kms-dt owner=magnum,
renewer=yarn, realUser=, issueDate=1598411250569, maxDate=1599016050569,
sequenceNumber=2791602, masterKeyId=908)
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind:
HDFS_DELEGATION_TOKEN, Service: ha-hdfs:at, Ident: (token for magnum:
HDFS_DELEGATION_TOKEN owner=mag...@test.com, renewer=yarn, realUser=,
issueDate=1598411249950, maxDate=1599275249950, sequenceNumber=1359401,
masterKeyId=461)
20/08/26 12:07:33 INFO impl.NMClientAsyncImpl: Upper bound of the thread pool
size is 500
20/08/26 12:07:33 WARN distributedshell.ApplicationMaster: Timeline service is
not enabled
{noformat}
DOCKER_CLIENT_CREDENTIAL_TOKEN is not propagated to the container.
eventually docker container launch that requires auth token failed.
was:
Job sumbit client side log
{noformat}
2020-08-26 12:07:30,756 INFO distributedshell.Client: Got dt for hdfs://at;
Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:at, Ident: (token for magnum:
HDFS_DELEGATION_TOKEN owner=mag...@test.com, renewer=yarn, realUser=,
issueDate=1598411249950, maxDate=1599275249950, sequenceNumber=1359401,
masterKeyId=461)
2020-08-26 12:07:30,757 INFO distributedshell.Client: Got dt for hdfs://at;
Kind: kms-dt, Service: kms://ht...@ranger-kms.test.com:9393/kms, Ident: (kms-dt
owner=magnum, renewer=yarn, realUser=, issueDate=1598411250569,
maxDate=1599016050569, sequenceNumber=2791602, masterKeyId=908)
2020-08-26 12:07:30,969 INFO util.DockerClientConfigHandler: Token read from
Docker client configuration file: Kind: DOCKER_CLIENT_CREDENTIAL_TOKEN,
Service: reg.test.com, Ident: (registryUrl: "reg.test.com" applicationId:
"application_1593772175687_475926")
{noformat}
AM LOG
{noformat}
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Executing with
tokens:
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind:
YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 475926
cluster_timestamp: 1593772175687 } attemptId: 1 } keyId: 1589057158)
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind: kms-dt,
Service: kms://ht...@ranger-kms.test.com:9393/kms, Ident: (kms-dt owner=magnum,
renewer=yarn, realUser=, issueDate=1598411250569, maxDate=1599016050569,
sequenceNumber=2791602, masterKeyId=908)
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind:
HDFS_DELEGATION_TOKEN, Service: ha-hdfs:at, Ident: (token for magnum:
HDFS_DELEGATION_TOKEN owner=mag...@test.com, renewer=yarn, realUser=,
issueDate=1598411249950, maxDate=1599275249950, sequenceNumber=1359401,
masterKeyId=461)
20/08/26 12:07:33 INFO impl.NMClientAsyncImpl: Upper bound of the thread pool
size is 500
20/08/26 12:07:33 WARN distributedshell.ApplicationMaster: Timeline service is
not enabled
{noformat}
docker client config
DOCKER_CLIENT_CREDENTIAL_TOKEN is not propagated to the container.
> Does not work specifying the docker client configuration in distributed shell
> -
>
> Key: YARN-10410
> URL: https://issues.apache.org/jira/browse/YARN-10410
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications/distributed-shell
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Priority: Major
>
> Job sumbit client side log
>
[jira] [Created] (YARN-10410) Does not work specifying the docker client configuration in distributed shell
KWON BYUNGCHANG created YARN-10410:
--
Summary: Does not work specifying the docker client configuration
in distributed shell
Key: YARN-10410
URL: https://issues.apache.org/jira/browse/YARN-10410
Project: Hadoop YARN
Issue Type: Bug
Components: applications/distributed-shell
Affects Versions: 3.1.2
Reporter: KWON BYUNGCHANG
Job sumbit client side log
{noformat}
2020-08-26 12:07:30,756 INFO distributedshell.Client: Got dt for hdfs://at;
Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:at, Ident: (token for magnum:
HDFS_DELEGATION_TOKEN owner=mag...@test.com, renewer=yarn, realUser=,
issueDate=1598411249950, maxDate=1599275249950, sequenceNumber=1359401,
masterKeyId=461)
2020-08-26 12:07:30,757 INFO distributedshell.Client: Got dt for hdfs://at;
Kind: kms-dt, Service: kms://ht...@ranger-kms.test.com:9393/kms, Ident: (kms-dt
owner=magnum, renewer=yarn, realUser=, issueDate=1598411250569,
maxDate=1599016050569, sequenceNumber=2791602, masterKeyId=908)
2020-08-26 12:07:30,969 INFO util.DockerClientConfigHandler: Token read from
Docker client configuration file: Kind: DOCKER_CLIENT_CREDENTIAL_TOKEN,
Service: reg.test.com, Ident: (registryUrl: "reg.test.com" applicationId:
"application_1593772175687_475926")
{noformat}
AM LOG
{noformat}
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Executing with
tokens:
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind:
YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 475926
cluster_timestamp: 1593772175687 } attemptId: 1 } keyId: 1589057158)
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind: kms-dt,
Service: kms://ht...@ranger-kms.test.com:9393/kms, Ident: (kms-dt owner=magnum,
renewer=yarn, realUser=, issueDate=1598411250569, maxDate=1599016050569,
sequenceNumber=2791602, masterKeyId=908)
20/08/26 12:07:33 INFO distributedshell.ApplicationMaster: Kind:
HDFS_DELEGATION_TOKEN, Service: ha-hdfs:at, Ident: (token for magnum:
HDFS_DELEGATION_TOKEN owner=mag...@test.com, renewer=yarn, realUser=,
issueDate=1598411249950, maxDate=1599275249950, sequenceNumber=1359401,
masterKeyId=461)
20/08/26 12:07:33 INFO impl.NMClientAsyncImpl: Upper bound of the thread pool
size is 500
20/08/26 12:07:33 WARN distributedshell.ApplicationMaster: Timeline service is
not enabled
{noformat}
docker client config
DOCKER_CLIENT_CREDENTIAL_TOKEN is not propagated to the container.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated YARN-9731:
--
Attachment: YARN-9731.005.patch
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, YARN-9731.002.patch,
> YARN-9731.003.patch, YARN-9731.004.patch, YARN-9731.005.patch,
> ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904882#comment-16904882
]
KWON BYUNGCHANG commented on YARN-9731:
---
[~abmodi] thank you for review. I've attched patch.
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, YARN-9731.002.patch,
> YARN-9731.003.patch, YARN-9731.004.patch, YARN-9731.005.patch,
> ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated YARN-9731:
--
Attachment: YARN-9731.004.patch
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, YARN-9731.002.patch,
> YARN-9731.003.patch, YARN-9731.004.patch, ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated YARN-9731:
--
Attachment: YARN-9731.003.patch
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, YARN-9731.002.patch,
> YARN-9731.003.patch, ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903706#comment-16903706
]
KWON BYUNGCHANG edited comment on YARN-9731 at 8/9/19 8:38 AM:
---
[~Prabhu Joseph] I've attached patch fixed testcases.
was (Author: magnum):
[~Prabhu Joseph] I've attaced patch fixed testcases.
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, YARN-9731.002.patch,
> ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated YARN-9731:
--
Attachment: YARN-9731.002.patch
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, YARN-9731.002.patch,
> ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903706#comment-16903706
]
KWON BYUNGCHANG commented on YARN-9731:
---
[~Prabhu Joseph] I've attaced patch fixed testcases.
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, YARN-9731.002.patch,
> ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903585#comment-16903585
]
KWON BYUNGCHANG commented on YARN-9731:
---
[~Prabhu Joseph] Thank you. I changed to status of patch avaiable.
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9732) yarn.system-metrics-publisher.enabled=false does not work
[ https://issues.apache.org/jira/browse/YARN-9732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-9732: -- Attachment: YARN-9732.0001.patch > yarn.system-metrics-publisher.enabled=false does not work > - > > Key: YARN-9732 > URL: https://issues.apache.org/jira/browse/YARN-9732 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineclient >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Priority: Major > Attachments: YARN-9732.0001.patch > > > RM does not use yarn.system-metrics-publisher.enabled=false, > so if configure only yarn.timeline-service.enabled=true, > YARN system metrics are always published on the timeline server by RM > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9732) yarn.system-metrics-publisher.enabled=false does not work
KWON BYUNGCHANG created YARN-9732: - Summary: yarn.system-metrics-publisher.enabled=false does not work Key: YARN-9732 URL: https://issues.apache.org/jira/browse/YARN-9732 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, timelineclient Affects Versions: 3.1.2 Reporter: KWON BYUNGCHANG RM does not use yarn.system-metrics-publisher.enabled=false, so if configure only yarn.timeline-service.enabled=true, YARN system metrics are always published on the timeline server by RM -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated YARN-9731:
--
Attachment: YARN-9731.001.patch
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
KWON BYUNGCHANG created YARN-9731:
-
Summary: In ATS v1.5, all jobs are visible to all users without
view-acl
Key: YARN-9731
URL: https://issues.apache.org/jira/browse/YARN-9731
Project: Hadoop YARN
Issue Type: Bug
Components: timelineserver
Affects Versions: 3.1.2
Reporter: KWON BYUNGCHANG
Attachments: ats_v1.5_screenshot.png
In ATS v1.5 of secure mode,
all jobs are visible to all users without view-acl.
if user does not have view-acl, user should not be able to see jobs.
I attatched ATS UI screenshot.
ATS v1.5 log
{code:java}
2019-08-09 10:21:13,679 WARN
applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
(ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
- Failed to authorize when generating application report for
application_1565247558150_1954. Use a placeholder for its latest attempt id.
org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
not have privilege to see this application application_1565247558150_1954
2019-08-09 10:21:13,680 WARN
applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
(ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
- Failed to authorize when generating application report for
application_1565247558150_1951. Use a placeholder for its latest attempt id.
org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
not have privilege to see this application application_1565247558150_1951
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9583) Failed job which is submitted unknown queue is showed all users
[ https://issues.apache.org/jira/browse/YARN-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900622#comment-16900622 ] KWON BYUNGCHANG commented on YARN-9583: --- [~sunilg] thank you for feedback. I checked as you mentioned. RM recovered app as SUCCESS. This patch does not changed App state. It just changed checking of permission. Need test code? > Failed job which is submitted unknown queue is showed all users > --- > > Key: YARN-9583 > URL: https://issues.apache.org/jira/browse/YARN-9583 > Project: Hadoop YARN > Issue Type: Bug > Components: security >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Assignee: KWON BYUNGCHANG >Priority: Major > Attachments: YARN-9583-screenshot.png, YARN-9583.001.patch, > YARN-9583.002.patch, YARN-9583.003.patch, YARN-9583.004.patch, > YARN-9583.005.patch > > > In secure mode, Failed job which is submitted unknown queue is showed all > users. > I attached RM UI screen shot. > reproduction senario >1. user foo submit job to unknown queue without view-acl and job will fail > immediately. >2. user bar can access the job of user foo which previously failed. > According to comments in QueueACLsManager .java that caused the problem, > This situation can happen when RM is restarted after deleting queue. > I think showing app of non existing queue to all users is the problem after > RM start. > It will become a security hole. > I fixed it a little bit. > After fixing it, Both owner of job and admin of yarn can access job which is > submitted unknown queue. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9647) Docker launch fails when local-dirs or log-dirs is unhealthy.
[
https://issues.apache.org/jira/browse/YARN-9647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885995#comment-16885995
]
KWON BYUNGCHANG commented on YARN-9647:
---
[~ebadger] Thanks for your comments.
The process of mounting volume in YARN is as follows.
step1. validate mountable point (docker.allowed.ro-mounts,
docker.allowed.rw-mounts) that is configured by yarn administrator in
/etc/hadoop/conf/container-executor.cfg
step2. validate mount point that is configured by user
step3. validate mount point of step2 belong to mountable point of step1
if /data2/ is unhealthy, threre is not /data2/ in mount point configuration
(step2) because nodemanager already know /data2 is unhealthy.
problem is /data2 still exists in /etc/hadoop/conf/container-executor.cfg
because container-exector.cfg is static configuation file.
and docker launch fails in step1 because container-executor cannot resolve real
path of /data2.
I simply modified step1 to ignore unresolving mountable path.
> Docker launch fails when local-dirs or log-dirs is unhealthy.
> -
>
> Key: YARN-9647
> URL: https://issues.apache.org/jira/browse/YARN-9647
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9647.001.patch, YARN-9647.002.patch
>
>
> my /etc/hadoop/conf/container-executor.cfg
> {code}
> [docker]
>docker.allowed.ro-mounts=/data1/hadoop/yarn/local,/data2/hadoop/yarn/local
>docker.allowed.rw-mounts=/data1/hadoop/yarn/local,/data2/hadoop/yarn/local
> {code}
> if /data2 is unhealthy, docker launch fails although container can use
> /data1 as local-dir, log-dir
> error message is below
> {code}
> [2019-06-25 14:55:26.168]Exception from container-launch. Container id:
> container_e50_1561100493387_5185_01_000597 Exit code: 29 Exception message:
> Launch container failed Shell error output: Could not determine real path of
> mount '/data2/hadoop/yarn/local' Could not determine real path of mount
> '/data2/hadoop/yarn/local' Unable to find permitted docker mounts on disk
> Error constructing docker command, docker error code=16, error message='Mount
> access error' Shell output: main : command provided 4 main : run as user is
> magnum main : requested yarn user is magnum Creating script paths... Creating
> local dirs... [2019-06-25 14:55:26.189]Container exited with a non-zero exit
> code 29. [2019-06-25 14:55:26.192]Container exited with a non-zero exit code
> 29.
> {code}
> root cause is that normalize_mounts() in docker-util.c return -1 because it
> cannot resolve real path of /data2/hadoop/yarn/local.(note that /data2 is
> disk fault at this point)
> however disk of nm local dirs and nm log dirs can fail at any time.
> docker launch should succeed if there are available local dirs and log dirs.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9647) Docker launch fails when local-dirs or log-dirs is unhealthy.
[
https://issues.apache.org/jira/browse/YARN-9647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated YARN-9647:
--
Attachment: YARN-9647.002.patch
> Docker launch fails when local-dirs or log-dirs is unhealthy.
> -
>
> Key: YARN-9647
> URL: https://issues.apache.org/jira/browse/YARN-9647
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9647.001.patch, YARN-9647.002.patch
>
>
> my /etc/hadoop/conf/container-executor.cfg
> {code}
> [docker]
>docker.allowed.ro-mounts=/data1/hadoop/yarn/local,/data2/hadoop/yarn/local
>docker.allowed.rw-mounts=/data1/hadoop/yarn/local,/data2/hadoop/yarn/local
> {code}
> if /data2 is unhealthy, docker launch fails although container can use
> /data1 as local-dir, log-dir
> error message is below
> {code}
> [2019-06-25 14:55:26.168]Exception from container-launch. Container id:
> container_e50_1561100493387_5185_01_000597 Exit code: 29 Exception message:
> Launch container failed Shell error output: Could not determine real path of
> mount '/data2/hadoop/yarn/local' Could not determine real path of mount
> '/data2/hadoop/yarn/local' Unable to find permitted docker mounts on disk
> Error constructing docker command, docker error code=16, error message='Mount
> access error' Shell output: main : command provided 4 main : run as user is
> magnum main : requested yarn user is magnum Creating script paths... Creating
> local dirs... [2019-06-25 14:55:26.189]Container exited with a non-zero exit
> code 29. [2019-06-25 14:55:26.192]Container exited with a non-zero exit code
> 29.
> {code}
> root cause is that normalize_mounts() in docker-util.c return -1 because it
> cannot resolve real path of /data2/hadoop/yarn/local.(note that /data2 is
> disk fault at this point)
> however disk of nm local dirs and nm log dirs can fail at any time.
> docker launch should succeed if there are available local dirs and log dirs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9647) Docker launch fails when local-dirs or log-dirs is unhealthy.
[
https://issues.apache.org/jira/browse/YARN-9647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated YARN-9647:
--
Affects Version/s: 3.1.2
> Docker launch fails when local-dirs or log-dirs is unhealthy.
> -
>
> Key: YARN-9647
> URL: https://issues.apache.org/jira/browse/YARN-9647
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Priority: Major
>
> my /etc/hadoop/conf/container-executor.cfg
> {code}
> [docker]
>docker.allowed.ro-mounts=/data1/hadoop/yarn/local,/data2/hadoop/yarn/local
>docker.allowed.rw-mounts=/data1/hadoop/yarn/local,/data2/hadoop/yarn/local
> {code}
> if /data2 is unhealthy, docker launch fails although container can use
> /data1 as local-dir, log-dir
> error message is below
> {code}
> [2019-06-25 14:55:26.168]Exception from container-launch. Container id:
> container_e50_1561100493387_5185_01_000597 Exit code: 29 Exception message:
> Launch container failed Shell error output: Could not determine real path of
> mount '/data2/hadoop/yarn/local' Could not determine real path of mount
> '/data2/hadoop/yarn/local' Unable to find permitted docker mounts on disk
> Error constructing docker command, docker error code=16, error message='Mount
> access error' Shell output: main : command provided 4 main : run as user is
> magnum main : requested yarn user is magnum Creating script paths... Creating
> local dirs... [2019-06-25 14:55:26.189]Container exited with a non-zero exit
> code 29. [2019-06-25 14:55:26.192]Container exited with a non-zero exit code
> 29.
> {code}
> root cause is that normalize_mounts() in docker-util.c return -1 because it
> cannot resolve real path of /data2/hadoop/yarn/local.(note that /data2 is
> disk fault at this point)
> however disk of nm local dirs and nm log dirs can fail at any time.
> docker launch should succeed if there are available local dirs and log dirs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9647) Docker launch fails when local-dirs or log-dirs is unhealthy.
KWON BYUNGCHANG created YARN-9647:
-
Summary: Docker launch fails when local-dirs or log-dirs is
unhealthy.
Key: YARN-9647
URL: https://issues.apache.org/jira/browse/YARN-9647
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Reporter: KWON BYUNGCHANG
my /etc/hadoop/conf/container-executor.cfg
{code}
[docker]
docker.allowed.ro-mounts=/data1/hadoop/yarn/local,/data2/hadoop/yarn/local
docker.allowed.rw-mounts=/data1/hadoop/yarn/local,/data2/hadoop/yarn/local
{code}
if /data2 is unhealthy, docker launch fails although container can use /data1
as local-dir, log-dir
error message is below
{code}
[2019-06-25 14:55:26.168]Exception from container-launch. Container id:
container_e50_1561100493387_5185_01_000597 Exit code: 29 Exception message:
Launch container failed Shell error output: Could not determine real path of
mount '/data2/hadoop/yarn/local' Could not determine real path of mount
'/data2/hadoop/yarn/local' Unable to find permitted docker mounts on disk Error
constructing docker command, docker error code=16, error message='Mount access
error' Shell output: main : command provided 4 main : run as user is magnum
main : requested yarn user is magnum Creating script paths... Creating local
dirs... [2019-06-25 14:55:26.189]Container exited with a non-zero exit code 29.
[2019-06-25 14:55:26.192]Container exited with a non-zero exit code 29.
{code}
root cause is that normalize_mounts() in docker-util.c return -1 because it
cannot resolve real path of /data2/hadoop/yarn/local.(note that /data2 is disk
fault at this point)
however disk of nm local dirs and nm log dirs can fail at any time.
docker launch should succeed if there are available local dirs and log dirs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9633) Support doas parameter at rest api of yarn-service
[ https://issues.apache.org/jira/browse/YARN-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-9633: -- Attachment: YARN-9633.002.patch > Support doas parameter at rest api of yarn-service > -- > > Key: YARN-9633 > URL: https://issues.apache.org/jira/browse/YARN-9633 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-native-services >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Priority: Major > Attachments: YARN-9633.001.patch, YARN-9633.002.patch > > > user can submit application of yarn-service with a more user-friendly web-ui > than RM UI2 > web-ui need proxy user privileges and the web-ui must be able to submit jobs > as an end user. > REST API of yarn-service need to doas function. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9633) Support doas parameter at rest api of yarn-service
[ https://issues.apache.org/jira/browse/YARN-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869072#comment-16869072 ] KWON BYUNGCHANG commented on YARN-9633: --- [~eyang] thank you for your feedback > Support doas parameter at rest api of yarn-service > -- > > Key: YARN-9633 > URL: https://issues.apache.org/jira/browse/YARN-9633 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-native-services >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Priority: Major > Attachments: YARN-9633.001.patch > > > user can submit application of yarn-service with a more user-friendly web-ui > than RM UI2 > web-ui need proxy user privileges and the web-ui must be able to submit jobs > as an end user. > REST API of yarn-service need to doas function. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9633) Support doas parameter at rest api of yarn-service
KWON BYUNGCHANG created YARN-9633: - Summary: Support doas parameter at rest api of yarn-service Key: YARN-9633 URL: https://issues.apache.org/jira/browse/YARN-9633 Project: Hadoop YARN Issue Type: Improvement Components: yarn-native-services Affects Versions: 3.1.2 Reporter: KWON BYUNGCHANG user can submit application of yarn-service with a more user-friendly web-ui than RM UI2 web-ui need proxy user privileges and the web-ui must be able to submit jobs as an end user. REST API of yarn-service need to doas function. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9583) Failed job which is submitted unknown queue is showed all users
[ https://issues.apache.org/jira/browse/YARN-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-9583: -- Priority: Major (was: Minor) > Failed job which is submitted unknown queue is showed all users > --- > > Key: YARN-9583 > URL: https://issues.apache.org/jira/browse/YARN-9583 > Project: Hadoop YARN > Issue Type: Bug > Components: security >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Priority: Major > Attachments: YARN-9583-screenshot.png, YARN-9583.001.patch, > YARN-9583.002.patch, YARN-9583.003.patch, YARN-9583.004.patch, > YARN-9583.005.patch > > > In secure mode, Failed job which is submitted unknown queue is showed all > users. > I attached RM UI screen shot. > reproduction senario >1. user foo submit job to unknown queue without view-acl and job will fail > immediately. >2. user bar can access the job of user foo which previously failed. > According to comments in QueueACLsManager .java that caused the problem, > This situation can happen when RM is restarted after deleting queue. > I think showing app of non existing queue to all users is the problem after > RM start. > It will become a security hole. > I fixed it a little bit. > After fixing it, Both owner of job and admin of yarn can access job which is > submitted unknown queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9583) Failed job which is submitted unknown queue is showed all users
[ https://issues.apache.org/jira/browse/YARN-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-9583: -- Attachment: YARN-9583.005.patch > Failed job which is submitted unknown queue is showed all users > --- > > Key: YARN-9583 > URL: https://issues.apache.org/jira/browse/YARN-9583 > Project: Hadoop YARN > Issue Type: Bug > Components: security >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Priority: Minor > Attachments: YARN-9583-screenshot.png, YARN-9583.001.patch, > YARN-9583.002.patch, YARN-9583.003.patch, YARN-9583.004.patch, > YARN-9583.005.patch > > > In secure mode, Failed job which is submitted unknown queue is showed all > users. > I attached RM UI screen shot. > reproduction senario >1. user foo submit job to unknown queue without view-acl and job will fail > immediately. >2. user bar can access the job of user foo which previously failed. > According to comments in QueueACLsManager .java that caused the problem, > This situation can happen when RM is restarted after deleting queue. > I think showing app of non existing queue to all users is the problem after > RM start. > It will become a security hole. > I fixed it a little bit. > After fixing it, Both owner of job and admin of yarn can access job which is > submitted unknown queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9583) Failed job which is submitted unknown queue is showed all users
[ https://issues.apache.org/jira/browse/YARN-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849453#comment-16849453 ] KWON BYUNGCHANG commented on YARN-9583: --- I attached patch that modified checkstyle. review please. > Failed job which is submitted unknown queue is showed all users > --- > > Key: YARN-9583 > URL: https://issues.apache.org/jira/browse/YARN-9583 > Project: Hadoop YARN > Issue Type: Bug > Components: security >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Priority: Minor > Attachments: YARN-9583-screenshot.png, YARN-9583.001.patch, > YARN-9583.002.patch, YARN-9583.003.patch, YARN-9583.004.patch > > > In secure mode, Failed job which is submitted unknown queue is showed all > users. > I attached RM UI screen shot. > reproduction senario >1. user foo submit job to unknown queue without view-acl and job will fail > immediately. >2. user bar can access the job of user foo which previously failed. > According to comments in QueueACLsManager .java that caused the problem, > This situation can happen when RM is restarted after deleting queue. > I think showing app of non existing queue to all users is the problem after > RM start. > It will become a security hole. > I fixed it a little bit. > After fixing it, Both owner of job and admin of yarn can access job which is > submitted unknown queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9583) Failed job which is submitted unknown queue is showed all users
[ https://issues.apache.org/jira/browse/YARN-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-9583: -- Attachment: YARN-9583.004.patch > Failed job which is submitted unknown queue is showed all users > --- > > Key: YARN-9583 > URL: https://issues.apache.org/jira/browse/YARN-9583 > Project: Hadoop YARN > Issue Type: Bug > Components: security >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Priority: Minor > Attachments: YARN-9583-screenshot.png, YARN-9583.001.patch, > YARN-9583.002.patch, YARN-9583.003.patch, YARN-9583.004.patch > > > In secure mode, Failed job which is submitted unknown queue is showed all > users. > I attached RM UI screen shot. > reproduction senario >1. user foo submit job to unknown queue without view-acl and job will fail > immediately. >2. user bar can access the job of user foo which previously failed. > According to comments in QueueACLsManager .java that caused the problem, > This situation can happen when RM is restarted after deleting queue. > I think showing app of non existing queue to all users is the problem after > RM start. > It will become a security hole. > I fixed it a little bit. > After fixing it, Both owner of job and admin of yarn can access job which is > submitted unknown queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9583) Failed job which is submitted unknown queue is showed all users
[ https://issues.apache.org/jira/browse/YARN-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849350#comment-16849350 ] KWON BYUNGCHANG commented on YARN-9583: --- I attached patch with test code. please review it. > Failed job which is submitted unknown queue is showed all users > --- > > Key: YARN-9583 > URL: https://issues.apache.org/jira/browse/YARN-9583 > Project: Hadoop YARN > Issue Type: Bug > Components: security >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Priority: Minor > Attachments: YARN-9583-screenshot.png, YARN-9583.001.patch, > YARN-9583.002.patch, YARN-9583.003.patch > > > In secure mode, Failed job which is submitted unknown queue is showed all > users. > I attached RM UI screen shot. > reproduction senario >1. user foo submit job to unknown queue without view-acl and job will fail > immediately. >2. user bar can access the job of user foo which previously failed. > According to comments in QueueACLsManager .java that caused the problem, > This situation can happen when RM is restarted after deleting queue. > I think showing app of non existing queue to all users is the problem after > RM start. > It will become a security hole. > I fixed it a little bit. > After fixing it, Both owner of job and admin of yarn can access job which is > submitted unknown queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9583) Failed job which is submitted unknown queue is showed all users
[ https://issues.apache.org/jira/browse/YARN-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-9583: -- Attachment: YARN-9583.003.patch > Failed job which is submitted unknown queue is showed all users > --- > > Key: YARN-9583 > URL: https://issues.apache.org/jira/browse/YARN-9583 > Project: Hadoop YARN > Issue Type: Bug > Components: security >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Priority: Minor > Attachments: YARN-9583-screenshot.png, YARN-9583.001.patch, > YARN-9583.002.patch, YARN-9583.003.patch > > > In secure mode, Failed job which is submitted unknown queue is showed all > users. > I attached RM UI screen shot. > reproduction senario >1. user foo submit job to unknown queue without view-acl and job will fail > immediately. >2. user bar can access the job of user foo which previously failed. > According to comments in QueueACLsManager .java that caused the problem, > This situation can happen when RM is restarted after deleting queue. > I think showing app of non existing queue to all users is the problem after > RM start. > It will become a security hole. > I fixed it a little bit. > After fixing it, Both owner of job and admin of yarn can access job which is > submitted unknown queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9583) Failed job which is submitted unknown queue is showed all users
[ https://issues.apache.org/jira/browse/YARN-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-9583: -- Attachment: YARN-9583.002.patch > Failed job which is submitted unknown queue is showed all users > --- > > Key: YARN-9583 > URL: https://issues.apache.org/jira/browse/YARN-9583 > Project: Hadoop YARN > Issue Type: Bug > Components: security >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Priority: Minor > Attachments: YARN-9583-screenshot.png, YARN-9583.001.patch, > YARN-9583.002.patch > > > In secure mode, Failed job which is submitted unknown queue is showed all > users. > I attached RM UI screen shot. > reproduction senario >1. user foo submit job to unknown queue without view-acl and job will fail > immediately. >2. user bar can access the job of user foo which previously failed. > According to comments in QueueACLsManager .java that caused the problem, > This situation can happen when RM is restarted after deleting queue. > I think showing app of non existing queue to all users is the problem after > RM start. > It will become a security hole. > I fixed it a little bit. > After fixing it, Both owner of job and admin of yarn can access job which is > submitted unknown queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9583) Failed job which is submitted unknown queue is showed all users
[ https://issues.apache.org/jira/browse/YARN-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-9583: -- Description: In secure mode, Failed job which is submitted unknown queue is showed all users. I attached RM UI screen shot. reproduction senario 1. user foo submit job to unknown queue without view-acl and job will fail immediately. 2. user bar can access the job of user foo which previously failed. According to comments in QueueACLsManager .java that caused the problem, This situation can happen when RM is restarted after deleting queue. I think showing app of non existing queue to all users is the problem after RM start. It will become a security hole. I fixed it a little bit. After fixing it, Both owner of job and admin of yarn can access job which is submitted unknown queue. was: In secure mode, Failed job which is submitted unknown queue is showed all users. I attached RM UI screen shot. reproduction senario 1. user foo submit job to unknown queue without view-acl and job will fail immediately. 2. user bar can access the job of user foo which previously failed. According to comments in QueueACLsManager .java that caused the problem, This situation can happen when RM is restarted after deletion queue. I think showing app of non existing queue to all users is the problem after RM start. It will become a security hole. I fixed it a little bit. After fixing it, Both owner of job and admin of yarn can access job which is submitted unknown queue. > Failed job which is submitted unknown queue is showed all users > --- > > Key: YARN-9583 > URL: https://issues.apache.org/jira/browse/YARN-9583 > Project: Hadoop YARN > Issue Type: Bug > Components: security >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Priority: Minor > Attachments: YARN-9583-screenshot.png, YARN-9583.001.patch > > > In secure mode, Failed job which is submitted unknown queue is showed all > users. > I attached RM UI screen shot. > reproduction senario >1. user foo submit job to unknown queue without view-acl and job will fail > immediately. >2. user bar can access the job of user foo which previously failed. > According to comments in QueueACLsManager .java that caused the problem, > This situation can happen when RM is restarted after deleting queue. > I think showing app of non existing queue to all users is the problem after > RM start. > It will become a security hole. > I fixed it a little bit. > After fixing it, Both owner of job and admin of yarn can access job which is > submitted unknown queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9583) Failed job which is submitted unknown queue is showed all users
[ https://issues.apache.org/jira/browse/YARN-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-9583: -- Description: In secure mode, Failed job which is submitted unknown queue is showed all users. I attached RM UI screen shot. reproduction senario 1. user foo submit job to unknown queue without view-acl and job will fail immediately. 2. user bar can access the job of user foo which previously failed. According to comments in QueueACLsManager .java that caused the problem, This situation can happen when RM is restarted after deletion queue. I think showing app of non existing queue to all users is the problem after RM start. It will become a security hole. I fixed it a little bit. After fixing it, Both owner of job and admin of yarn can access job which is submitted unknown queue. was: Failed job which is submitted unknown queue is showed all users. I attached RM UI screen shot. reproduction senario 1. user foo submit job to unknown queue without view-acl and job will fail immediately. 2. user bar can access the job of user foo which previously failed. According to comments in QueueACLsManager .java that caused the problem, This situation can happen when RM is restarted after deletion queue. I think showing app of non existing queue to all users is the problem after RM start. It will become a security hole. I fixed it a little bit. After fixing it, Both owner of job and admin of yarn can access job which is submitted unknown queue. > Failed job which is submitted unknown queue is showed all users > --- > > Key: YARN-9583 > URL: https://issues.apache.org/jira/browse/YARN-9583 > Project: Hadoop YARN > Issue Type: Bug > Components: security >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Priority: Minor > Attachments: YARN-9583-screenshot.png, YARN-9583.001.patch > > > In secure mode, Failed job which is submitted unknown queue is showed all > users. > I attached RM UI screen shot. > reproduction senario >1. user foo submit job to unknown queue without view-acl and job will fail > immediately. >2. user bar can access the job of user foo which previously failed. > According to comments in QueueACLsManager .java that caused the problem, > This situation can happen when RM is restarted after deletion queue. > I think showing app of non existing queue to all users is the problem after > RM start. > It will become a security hole. > I fixed it a little bit. > After fixing it, Both owner of job and admin of yarn can access job which is > submitted unknown queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9583) Failed job which is submitted unknown queue is showed all users
[ https://issues.apache.org/jira/browse/YARN-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-9583: -- Description: Failed job which is submitted unknown queue is showed all users. I attached RM UI screen shot. reproduction senario 1. user foo submit job to unknown queue without view-acl and job will fail immediately. 2. user bar can access the job of user foo which previously failed. According to comments in QueueACLsManager .java that caused the problem, This situation can happen when RM is restarted after deletion queue. I think showing app of non existing queue to all users is the problem after RM start. It will become a security hole. I fixed it a little bit. After fixing it, Both owner of job and admin of yarn can access job which is submitted unknown queue. was: Failed job which is submitted unknown queue is showed all users. I attached RM UI screen shot. reproduction senario 1. user foo submit job to unknown queue without view-acl and failed job. 2. user bar can access job of user foo. According to comments in QueueACLsManager .java that caused the problem. This situation can happen when RM is restarted after deletion queue. I think showing app of non existing queue to all users is the problem after RM start. It will become a security hole. I fixed it a little bit. After fixing it, Both owner of job and admin of yarn can access job which is submitted unknown queue. > Failed job which is submitted unknown queue is showed all users > --- > > Key: YARN-9583 > URL: https://issues.apache.org/jira/browse/YARN-9583 > Project: Hadoop YARN > Issue Type: Bug > Components: security >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Priority: Minor > Attachments: YARN-9583-screenshot.png, YARN-9583.001.patch > > > Failed job which is submitted unknown queue is showed all users. > I attached RM UI screen shot. > reproduction senario >1. user foo submit job to unknown queue without view-acl and job will fail > immediately. >2. user bar can access the job of user foo which previously failed. > According to comments in QueueACLsManager .java that caused the problem, > This situation can happen when RM is restarted after deletion queue. > I think showing app of non existing queue to all users is the problem after > RM start. > It will become a security hole. > I fixed it a little bit. > After fixing it, Both owner of job and admin of yarn can access job which is > submitted unknown queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9583) Failed job which is submitted unknown queue is showed all users
[ https://issues.apache.org/jira/browse/YARN-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-9583: -- Attachment: YARN-9583.001.patch > Failed job which is submitted unknown queue is showed all users > --- > > Key: YARN-9583 > URL: https://issues.apache.org/jira/browse/YARN-9583 > Project: Hadoop YARN > Issue Type: Bug > Components: security >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Priority: Minor > Attachments: YARN-9583-screenshot.png, YARN-9583.001.patch > > > Failed job which is submitted unknown queue is showed all users. > I attached RM UI screen shot. > reproduction senario >1. user foo submit job to unknown queue without view-acl and failed job. >2. user bar can access job of user foo. > According to comments in QueueACLsManager .java that caused the problem. > This situation can happen when RM is restarted after deletion queue. > I think showing app of non existing queue to all users is the problem after > RM start. > It will become a security hole. > I fixed it a little bit. > After fixing it, Both owner of job and admin of yarn can access job which is > submitted unknown queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9583) Failed job which is submitted unknown queue is showed all users
[ https://issues.apache.org/jira/browse/YARN-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KWON BYUNGCHANG updated YARN-9583: -- Attachment: YARN-9583-screenshot.png > Failed job which is submitted unknown queue is showed all users > --- > > Key: YARN-9583 > URL: https://issues.apache.org/jira/browse/YARN-9583 > Project: Hadoop YARN > Issue Type: Bug > Components: security >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Priority: Minor > Attachments: YARN-9583-screenshot.png > > > Failed job which is submitted unknown queue is showed all users. > I attached RM UI screen shot. > reproduction senario >1. user foo submit job to unknown queue without view-acl and failed job. >2. user bar can access job of user foo. > According to comments in QueueACLsManager .java that caused the problem. > This situation can happen when RM is restarted after deletion queue. > I think showing app of non existing queue to all users is the problem after > RM start. > It will become a security hole. > I fixed it a little bit. > After fixing it, Both owner of job and admin of yarn can access job which is > submitted unknown queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9583) Failed job which is submitted unknown queue is showed all users
KWON BYUNGCHANG created YARN-9583: - Summary: Failed job which is submitted unknown queue is showed all users Key: YARN-9583 URL: https://issues.apache.org/jira/browse/YARN-9583 Project: Hadoop YARN Issue Type: Bug Components: security Affects Versions: 3.1.2 Reporter: KWON BYUNGCHANG Failed job which is submitted unknown queue is showed all users. I attached RM UI screen shot. reproduction senario 1. user foo submit job to unknown queue without view-acl and failed job. 2. user bar can access job of user foo. According to comments in QueueACLsManager .java that caused the problem. This situation can happen when RM is restarted after deletion queue. I think showing app of non existing queue to all users is the problem after RM start. It will become a security hole. I fixed it a little bit. After fixing it, Both owner of job and admin of yarn can access job which is submitted unknown queue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8741) When multiple http filter is configured, RMAuthenticationFilterInitializer does not override
KWON BYUNGCHANG created YARN-8741:
-
Summary: When multiple http filter is configured,
RMAuthenticationFilterInitializer does not override
Key: YARN-8741
URL: https://issues.apache.org/jira/browse/YARN-8741
Project: Hadoop YARN
Issue Type: Bug
Reporter: KWON BYUNGCHANG
if security does not enabled and configure
hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter,
RMAuthenticationFilterInitializer is appended at that config key by
resourcemanager
RMWebAppUtil.java
{code:java}
String initializers = conf.get(filterInitializerConfKey);
if (!UserGroupInformation.isSecurityEnabled()) {
.
} else if (initializers.equals(StaticUserWebFilter.class.getName())) {
conf.set(filterInitializerConfKey,
RMAuthenticationFilterInitializer.class.getName() + ","
+ initializers);
conf.set(authTypeKey, "simple");
}
}
{code}
hadoop.http.filter.initializers can be configured multiple values and
when multiple http filters are configured (e.g. YARN-4009),
RMAuthenticationFilterInitializer does not override.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6711) log aggregation can't use other filesystem instead of fs.defaultFS
[
https://issues.apache.org/jira/browse/YARN-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG resolved YARN-6711.
---
Resolution: Duplicate
> log aggregation can't use other filesystem instead of fs.defaultFS
> --
>
> Key: YARN-6711
> URL: https://issues.apache.org/jira/browse/YARN-6711
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1, 3.0.0-alpha3
>Reporter: KWON BYUNGCHANG
>Priority: Minor
>
> hdfs support federated namespace.
> fs.defaultFS should choice one of federated namespace.
> however if {{yarn.nodemanager.remote-app-log-dir}} does not belong to default
> fs,
> nodemanager will stop.
> absolutely log-aggregation should can use other namespace instead of default
> fs.
> below is error log
> configuration is
> {quote}
> fs.defaultFS=hdfs://x3
> yarn.nodemanager.remote-app-log-dir=hdfs://x3v/logs
> {quote}
> {code}
> 2017-06-14 15:59:47,792 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher:
> Error in dispatcher thread
> java.lang.IllegalArgumentException: Wrong FS: hdfs://x3v/logs, expected:
> hdfs://x3
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:646)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:192)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:319)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:443)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:67)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
> at java.lang.Thread.run(Thread.java:745)
> 2017-06-14 15:59:47,794 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Exiting, bbye..
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6711) log aggregation can't use other filesystem instead of fs.defaultFS
[
https://issues.apache.org/jira/browse/YARN-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049876#comment-16049876
]
KWON BYUNGCHANG commented on YARN-6711:
---
thank you.
YARN-3269 figure out this issue.
close this as duplicate
> log aggregation can't use other filesystem instead of fs.defaultFS
> --
>
> Key: YARN-6711
> URL: https://issues.apache.org/jira/browse/YARN-6711
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1, 3.0.0-alpha3
>Reporter: KWON BYUNGCHANG
>Priority: Minor
>
> hdfs support federated namespace.
> fs.defaultFS should choice one of federated namespace.
> however if {{yarn.nodemanager.remote-app-log-dir}} does not belong to default
> fs,
> nodemanager will stop.
> absolutely log-aggregation should can use other namespace instead of default
> fs.
> below is error log
> configuration is
> {quote}
> fs.defaultFS=hdfs://x3
> yarn.nodemanager.remote-app-log-dir=hdfs://x3v/logs
> {quote}
> {code}
> 2017-06-14 15:59:47,792 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher:
> Error in dispatcher thread
> java.lang.IllegalArgumentException: Wrong FS: hdfs://x3v/logs, expected:
> hdfs://x3
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:646)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:192)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:319)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:443)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:67)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
> at java.lang.Thread.run(Thread.java:745)
> 2017-06-14 15:59:47,794 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Exiting, bbye..
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6711) log aggregation can't use other filesystem instead of fs.defaultFS
KWON BYUNGCHANG created YARN-6711:
-
Summary: log aggregation can't use other filesystem instead of
fs.defaultFS
Key: YARN-6711
URL: https://issues.apache.org/jira/browse/YARN-6711
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 3.0.0-alpha3, 2.7.1
Reporter: KWON BYUNGCHANG
Priority: Minor
hdfs support federated namespace.
fs.defaultFS should choice one of federated namespace.
however if {{yarn.nodemanager.remote-app-log-dir}} does not belong to default
fs,
nodemanager will stop.
absolutely log-aggregation should can use other namespace instead of default fs.
below is error log
configuration is
{quote}
fs.defaultFS=hdfs://x3
yarn.nodemanager.remote-app-log-dir=hdfs://x3v/logs
{quote}
{code}
2017-06-14 15:59:47,792 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher:
Error in dispatcher thread
java.lang.IllegalArgumentException: Wrong FS: hdfs://x3v/logs, expected:
hdfs://x3
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:646)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:192)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:319)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:443)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:67)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
at java.lang.Thread.run(Thread.java:745)
2017-06-14 15:59:47,794 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
Exiting, bbye..
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4666) API redirects from a standby resource manager drop query string parameters
[
https://issues.apache.org/jira/browse/YARN-4666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated YARN-4666:
--
Attachment: YARN-4666.patch
I've attached patch.
> API redirects from a standby resource manager drop query string parameters
> --
>
> Key: YARN-4666
> URL: https://issues.apache.org/jira/browse/YARN-4666
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.1
>Reporter: Antonio Bellezza
> Attachments: YARN-4666.patch
>
>
> When making an API request with a query string to a secondary resource
> manager, the redirection doesn't contain the query string.
> Example:
> {code}
> $ curl -v -H "Accept: application/json"
> "http://standby-server.mydomain:8088/ws/v1/cluster/apps?limit=10=someuser;
> * Hostname was NOT found in DNS cache
> * Trying 192.168.0.123...
> * Connected to standby-server.mydomain (192.168.0.123) port 8088 (#0)
> > GET /ws/v1/cluster/apps?limit=10=someuser HTTP/1.1
> > User-Agent: curl/7.35.0
> > Host: standby-server.mydomain:8088
> > Accept: application/json
> >
> < HTTP/1.1 307 TEMPORARY_REDIRECT
> < Cache-Control: no-cache
> < Expires: Fri, 22 Jan 2016 16:43:42 GMT
> < Date: Fri, 22 Jan 2016 16:43:42 GMT
> < Pragma: no-cache
> < Expires: Fri, 22 Jan 2016 16:43:42 GMT
> < Date: Fri, 22 Jan 2016 16:43:42 GMT
> < Pragma: no-cache
> < Content-Type: text/plain; charset=UTF-8
> < Location: http://active-server.mydomain:8088/ws/v1/cluster/apps
> < Content-Length: 105
> * Server Jetty(6.1.26.hwx) is not blacklisted
> < Server: Jetty(6.1.26.hwx)
> <
> This is standby RM. The redirect url is:
> http://active-server.mydomain:8088/ws/v1/cluster/apps
> * Connection #0 to host standby-server.mydomain left intact
> {code}
> This may depend on RMWebAppFilter generating the redirect path from
> request.getRequestURI(), which does not include query string parameters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4464) default value of yarn.resourcemanager.state-store.max-completed-applications should lower.
[
https://issues.apache.org/jira/browse/YARN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061968#comment-15061968
]
KWON BYUNGCHANG commented on YARN-4464:
---
I have a question.
my cluster is not secure mode.
{{RMAppRoot}} has 6 files. however {{RMDTSecretManagerRoot}} has 4020 files.
{code}
magnum-mbp:~/c3$ hdfs dfs -ls /system/yarn/rmstore/FSRMStateRoot/RMAppRoot | wc
-l
6
magnum-mbp:~/c3$ hdfs dfs -ls
/system/yarn/rmstore/FSRMStateRoot/RMDTSecretManagerRoot/ | wc -l
4020
{code}
I think files count of {{RMDTSecretManagerRoot}} will influence RM recovery
process.
How do I reduce files count?
I do not configure below properties.
{code}
yarn.resourcemanager.delegation.key.update-interval
yarn.resourcemanager.delegation.token.renew-interval
yarn.resourcemanager.delegation.token.max-lifetime
{code}
> default value of yarn.resourcemanager.state-store.max-completed-applications
> should lower.
> --
>
> Key: YARN-4464
> URL: https://issues.apache.org/jira/browse/YARN-4464
> Project: Hadoop YARN
> Issue Type: Wish
> Components: resourcemanager
>Reporter: KWON BYUNGCHANG
>Assignee: Daniel Templeton
>Priority: Blocker
>
> my cluster has 120 nodes.
> I configured RM Restart feature.
> {code}
> yarn.resourcemanager.recovery.enabled=true
> yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
> yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
> {code}
> unfortunately I did not configure
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> so that property configured default value 10,000.
> I have restarted RM due to changing another configuartion.
> I expected that RM restart immediately.
> recovery process was very slow. I have waited about 20min.
> realize missing
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> its default value is very huge.
> need to change lower value or document notice on [RM Restart
> page|http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4464) default value of yarn.resourcemanager.state-store.max-completed-applications should lower.
[
https://issues.apache.org/jira/browse/YARN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061481#comment-15061481
]
KWON BYUNGCHANG commented on YARN-4464:
---
0 is good :)
> default value of yarn.resourcemanager.state-store.max-completed-applications
> should lower.
> --
>
> Key: YARN-4464
> URL: https://issues.apache.org/jira/browse/YARN-4464
> Project: Hadoop YARN
> Issue Type: Wish
> Components: resourcemanager
>Reporter: KWON BYUNGCHANG
>Assignee: Daniel Templeton
>Priority: Blocker
>
> my cluster has 120 nodes.
> I configured RM Restart feature.
> {code}
> yarn.resourcemanager.recovery.enabled=true
> yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
> yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
> {code}
> unfortunately I did not configure
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> so that property configured default value 10,000.
> I have restarted RM due to changing another configuartion.
> I expected that RM restart immediately.
> recovery process was very slow. I have waited about 20min.
> realize missing
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> its default value is very huge.
> need to change lower value or document notice on [RM Restart
> page|http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-4464) default value of yarn.resourcemanager.state-store.max-completed-applications should lower.
KWON BYUNGCHANG created YARN-4464:
-
Summary: default value of
yarn.resourcemanager.state-store.max-completed-applications should lower.
Key: YARN-4464
URL: https://issues.apache.org/jira/browse/YARN-4464
Project: Hadoop YARN
Issue Type: Wish
Components: resourcemanager
Reporter: KWON BYUNGCHANG
my cluster has 120 nodes.
I configured RM Restart feature.
{code}
yarn.resourcemanager.recovery.enabled=true
yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
{code}
unfortunately I did not configure
{{yarn.resourcemanager.state-store.max-completed-applications}}.
so that property configured default value 10,000.
I have restarted RM due to changing another configuartion.
I expected that RM restart immediately.
recovery process was very slow. I have waited about 20min.
realize missing {{yarn.resourcemanager.state-store.max-completed-applications}}.
its default value is very huge.
need to change lower value or document notice on [RM Restart
page|http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
