[jira] [Updated] (YARN-9354) TestUtils#createResource calls should be replaced with ResourceTypesTestHelper#newResource

2020-03-05 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9354:
---
Attachment: YARN-9354.002.patch

> TestUtils#createResource calls should be replaced with 
> ResourceTypesTestHelper#newResource
> --
>
> Key: YARN-9354
> URL: https://issues.apache.org/jira/browse/YARN-9354
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Trivial
>  Labels: newbie, newbie++
> Attachments: YARN-9354.001.patch, YARN-9354.002.patch
>
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestUtils#createResource
>  has not identical, but very similar implementation to 
> org.apache.hadoop.yarn.resourcetypes.ResourceTypesTestHelper#newResource. 
> Since these 2 methods are doing the same essentially and 
> ResourceTypesTestHelper is newer and used more, TestUtils#createResource 
> should be replaced with ResourceTypesTestHelper#newResource with all 
> occurrence.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-9419) Log a warning if GPU isolation is enabled but LinuxContainerExecutor is disabled

2020-03-05 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori reassigned YARN-9419:
--

Assignee: Andras Gyori  (was: Kinga Marton)

> Log a warning if GPU isolation is enabled but LinuxContainerExecutor is 
> disabled
> 
>
> Key: YARN-9419
> URL: https://issues.apache.org/jira/browse/YARN-9419
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Major
>
> A WARN log should be added at least (logged once on startup) that notifies 
> the user about a potentially offending configuration: GPU isolation is 
> enabled but LCE is disabled.
> I think this is a dangerous, yet valid configuration: As LCE is the only 
> container executor that utilizes cgroups, no real HW-isolation happens if LCE 
> is disabled. 
> Let's suppose we have 2 GPU devices in 1 node:
>  # NM reports 2 devices (as a Resource) to RM
>  # RM assigns GPU#1 to container#2 that requests 1 GPU device
>  # When container#2 is also requesting 1 GPU device, RM is going to assign 
> either GPU#1 or GPU#2, so there's no guarantee that GPU#2 will be assigned. 
> If GPU#1 is assigned to a second container, nasty things could happen.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9419) Log a warning if GPU isolation is enabled but LinuxContainerExecutor is disabled

2020-03-06 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9419:
---
Attachment: YARN-9419.002.patch

> Log a warning if GPU isolation is enabled but LinuxContainerExecutor is 
> disabled
> 
>
> Key: YARN-9419
> URL: https://issues.apache.org/jira/browse/YARN-9419
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-9419.001.patch, YARN-9419.002.patch
>
>
> A WARN log should be added at least (logged once on startup) that notifies 
> the user about a potentially offending configuration: GPU isolation is 
> enabled but LCE is disabled.
> I think this is a dangerous, yet valid configuration: As LCE is the only 
> container executor that utilizes cgroups, no real HW-isolation happens if LCE 
> is disabled. 
> Let's suppose we have 2 GPU devices in 1 node:
>  # NM reports 2 devices (as a Resource) to RM
>  # RM assigns GPU#1 to container#2 that requests 1 GPU device
>  # When container#2 is also requesting 1 GPU device, RM is going to assign 
> either GPU#1 or GPU#2, so there's no guarantee that GPU#2 will be assigned. 
> If GPU#1 is assigned to a second container, nasty things could happen.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9419) Log a warning if GPU isolation is enabled but LinuxContainerExecutor is disabled

2020-03-06 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053129#comment-17053129
 ] 

Andras Gyori commented on YARN-9419:


I have taken the class-based equality comparison approach, because the other 
solution would lead to the alteration of different parts of the codebase, which 
might not belong to this issue in its entirety. 

> Log a warning if GPU isolation is enabled but LinuxContainerExecutor is 
> disabled
> 
>
> Key: YARN-9419
> URL: https://issues.apache.org/jira/browse/YARN-9419
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-9419.001.patch, YARN-9419.002.patch
>
>
> A WARN log should be added at least (logged once on startup) that notifies 
> the user about a potentially offending configuration: GPU isolation is 
> enabled but LCE is disabled.
> I think this is a dangerous, yet valid configuration: As LCE is the only 
> container executor that utilizes cgroups, no real HW-isolation happens if LCE 
> is disabled. 
> Let's suppose we have 2 GPU devices in 1 node:
>  # NM reports 2 devices (as a Resource) to RM
>  # RM assigns GPU#1 to container#2 that requests 1 GPU device
>  # When container#2 is also requesting 1 GPU device, RM is going to assign 
> either GPU#1 or GPU#2, so there's no guarantee that GPU#2 will be assigned. 
> If GPU#1 is assigned to a second container, nasty things could happen.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-9997) Code cleanup in ZKConfigurationStore

2020-03-05 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori reassigned YARN-9997:
--

Assignee: Andras Gyori

> Code cleanup in ZKConfigurationStore
> 
>
> Key: YARN-9997
> URL: https://issues.apache.org/jira/browse/YARN-9997
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Minor
>
> Many thins can be improved:
> * znodeParentPath could be a local variable
> * zkManager could be private, VisibleForTesting annotation is not needed 
> anymore
> * Do something with unchecked casts
> * zkManager.safeSetData calls are almost having the same set of parameters: 
> Simplify this
> * Extract zkManager calls to their own methods: They are repeated
> * Remove TODOs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9997) Code cleanup in ZKConfigurationStore

2020-03-10 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9997:
---
Attachment: YARN-9997.005.patch

> Code cleanup in ZKConfigurationStore
> 
>
> Key: YARN-9997
> URL: https://issues.apache.org/jira/browse/YARN-9997
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-9997.001.patch, YARN-9997.002.patch, 
> YARN-9997.003.patch, YARN-9997.004.patch, YARN-9997.005.patch
>
>
> Many thins can be improved:
> * znodeParentPath could be a local variable
> * zkManager could be private, VisibleForTesting annotation is not needed 
> anymore
> * Do something with unchecked casts
> * zkManager.safeSetData calls are almost having the same set of parameters: 
> Simplify this
> * Extract zkManager calls to their own methods: They are repeated
> * Remove TODOs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9354) Resources should be created with ResourceTypesTestHelper instead of TestUtils

2020-03-12 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9354:
---
Attachment: (was: YARN-9354.branch-3.2.002.patch)

> Resources should be created with ResourceTypesTestHelper instead of TestUtils
> -
>
> Key: YARN-9354
> URL: https://issues.apache.org/jira/browse/YARN-9354
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Trivial
>  Labels: newbie, newbie++
> Fix For: 3.3.0
>
> Attachments: YARN-9354.001.patch, YARN-9354.002.patch, 
> YARN-9354.003.patch, YARN-9354.004.patch, YARN-9354.branch-3.2.001.patch, 
> YARN-9354.branch-3.2.002.patch
>
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestUtils#createResource
>  has not identical, but very similar implementation to 
> org.apache.hadoop.yarn.resourcetypes.ResourceTypesTestHelper#newResource. 
> Since these 2 methods are doing the same essentially and 
> ResourceTypesTestHelper is newer and used more, TestUtils#createResource 
> should be replaced with ResourceTypesTestHelper#newResource with all 
> occurrence.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9354) Resources should be created with ResourceTypesTestHelper instead of TestUtils

2020-03-12 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9354:
---
Attachment: YARN-9354.branch-3.2.002.patch

> Resources should be created with ResourceTypesTestHelper instead of TestUtils
> -
>
> Key: YARN-9354
> URL: https://issues.apache.org/jira/browse/YARN-9354
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Trivial
>  Labels: newbie, newbie++
> Fix For: 3.3.0
>
> Attachments: YARN-9354.001.patch, YARN-9354.002.patch, 
> YARN-9354.003.patch, YARN-9354.004.patch, YARN-9354.branch-3.2.001.patch, 
> YARN-9354.branch-3.2.002.patch
>
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestUtils#createResource
>  has not identical, but very similar implementation to 
> org.apache.hadoop.yarn.resourcetypes.ResourceTypesTestHelper#newResource. 
> Since these 2 methods are doing the same essentially and 
> ResourceTypesTestHelper is newer and used more, TestUtils#createResource 
> should be replaced with ResourceTypesTestHelper#newResource with all 
> occurrence.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9997) Code cleanup in ZKConfigurationStore

2020-03-11 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057024#comment-17057024
 ] 

Andras Gyori commented on YARN-9997:


Thank you for notification [~snemeth],
The conflict has been resolved and could be successfully applied to the latest 
trunk revision.

> Code cleanup in ZKConfigurationStore
> 
>
> Key: YARN-9997
> URL: https://issues.apache.org/jira/browse/YARN-9997
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-9997.001.patch, YARN-9997.002.patch, 
> YARN-9997.003.patch, YARN-9997.004.patch, YARN-9997.005.patch, 
> YARN-9997.006.patch
>
>
> Many thins can be improved:
> * znodeParentPath could be a local variable
> * zkManager could be private, VisibleForTesting annotation is not needed 
> anymore
> * Do something with unchecked casts
> * zkManager.safeSetData calls are almost having the same set of parameters: 
> Simplify this
> * Extract zkManager calls to their own methods: They are repeated
> * Remove TODOs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9997) Code cleanup in ZKConfigurationStore

2020-03-11 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9997:
---
Attachment: YARN-9997.006.patch

> Code cleanup in ZKConfigurationStore
> 
>
> Key: YARN-9997
> URL: https://issues.apache.org/jira/browse/YARN-9997
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-9997.001.patch, YARN-9997.002.patch, 
> YARN-9997.003.patch, YARN-9997.004.patch, YARN-9997.005.patch, 
> YARN-9997.006.patch
>
>
> Many thins can be improved:
> * znodeParentPath could be a local variable
> * zkManager could be private, VisibleForTesting annotation is not needed 
> anymore
> * Do something with unchecked casts
> * zkManager.safeSetData calls are almost having the same set of parameters: 
> Simplify this
> * Extract zkManager calls to their own methods: They are repeated
> * Remove TODOs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9354) Resources should be created with ResourceTypesTestHelper instead of TestUtils

2020-03-11 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9354:
---
Attachment: YARN-9354.branch-3.2.001.patch

> Resources should be created with ResourceTypesTestHelper instead of TestUtils
> -
>
> Key: YARN-9354
> URL: https://issues.apache.org/jira/browse/YARN-9354
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Trivial
>  Labels: newbie, newbie++
> Fix For: 3.3.0
>
> Attachments: YARN-9354.001.patch, YARN-9354.002.patch, 
> YARN-9354.003.patch, YARN-9354.004.patch, YARN-9354.branch-3.2.001.patch
>
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestUtils#createResource
>  has not identical, but very similar implementation to 
> org.apache.hadoop.yarn.resourcetypes.ResourceTypesTestHelper#newResource. 
> Since these 2 methods are doing the same essentially and 
> ResourceTypesTestHelper is newer and used more, TestUtils#createResource 
> should be replaced with ResourceTypesTestHelper#newResource with all 
> occurrence.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9419) Log a warning if GPU isolation is enabled but LinuxContainerExecutor is disabled

2020-03-09 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054739#comment-17054739
 ] 

Andras Gyori commented on YARN-9419:


Thanks for the reviews, [~adam.antal].

I have applied your suggestions in the code.

 

> Log a warning if GPU isolation is enabled but LinuxContainerExecutor is 
> disabled
> 
>
> Key: YARN-9419
> URL: https://issues.apache.org/jira/browse/YARN-9419
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-9419.001.patch, YARN-9419.002.patch, 
> YARN-9419.003.patch
>
>
> A WARN log should be added at least (logged once on startup) that notifies 
> the user about a potentially offending configuration: GPU isolation is 
> enabled but LCE is disabled.
> I think this is a dangerous, yet valid configuration: As LCE is the only 
> container executor that utilizes cgroups, no real HW-isolation happens if LCE 
> is disabled. 
> Let's suppose we have 2 GPU devices in 1 node:
>  # NM reports 2 devices (as a Resource) to RM
>  # RM assigns GPU#1 to container#2 that requests 1 GPU device
>  # When container#2 is also requesting 1 GPU device, RM is going to assign 
> either GPU#1 or GPU#2, so there's no guarantee that GPU#2 will be assigned. 
> If GPU#1 is assigned to a second container, nasty things could happen.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9419) Log a warning if GPU isolation is enabled but LinuxContainerExecutor is disabled

2020-03-09 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9419:
---
Attachment: YARN-9419.003.patch

> Log a warning if GPU isolation is enabled but LinuxContainerExecutor is 
> disabled
> 
>
> Key: YARN-9419
> URL: https://issues.apache.org/jira/browse/YARN-9419
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-9419.001.patch, YARN-9419.002.patch, 
> YARN-9419.003.patch
>
>
> A WARN log should be added at least (logged once on startup) that notifies 
> the user about a potentially offending configuration: GPU isolation is 
> enabled but LCE is disabled.
> I think this is a dangerous, yet valid configuration: As LCE is the only 
> container executor that utilizes cgroups, no real HW-isolation happens if LCE 
> is disabled. 
> Let's suppose we have 2 GPU devices in 1 node:
>  # NM reports 2 devices (as a Resource) to RM
>  # RM assigns GPU#1 to container#2 that requests 1 GPU device
>  # When container#2 is also requesting 1 GPU device, RM is going to assign 
> either GPU#1 or GPU#2, so there's no guarantee that GPU#2 will be assigned. 
> If GPU#1 is assigned to a second container, nasty things could happen.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9997) Code cleanup in ZKConfigurationStore

2020-03-09 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9997:
---
Attachment: YARN-9997.003.patch

> Code cleanup in ZKConfigurationStore
> 
>
> Key: YARN-9997
> URL: https://issues.apache.org/jira/browse/YARN-9997
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-9997.001.patch, YARN-9997.002.patch, 
> YARN-9997.003.patch
>
>
> Many thins can be improved:
> * znodeParentPath could be a local variable
> * zkManager could be private, VisibleForTesting annotation is not needed 
> anymore
> * Do something with unchecked casts
> * zkManager.safeSetData calls are almost having the same set of parameters: 
> Simplify this
> * Extract zkManager calls to their own methods: They are repeated
> * Remove TODOs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9354) TestUtils#createResource calls should be replaced with ResourceTypesTestHelper#newResource

2020-03-09 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9354:
---
Attachment: YARN-9354.003.patch

> TestUtils#createResource calls should be replaced with 
> ResourceTypesTestHelper#newResource
> --
>
> Key: YARN-9354
> URL: https://issues.apache.org/jira/browse/YARN-9354
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Trivial
>  Labels: newbie, newbie++
> Attachments: YARN-9354.001.patch, YARN-9354.002.patch, 
> YARN-9354.003.patch
>
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestUtils#createResource
>  has not identical, but very similar implementation to 
> org.apache.hadoop.yarn.resourcetypes.ResourceTypesTestHelper#newResource. 
> Since these 2 methods are doing the same essentially and 
> ResourceTypesTestHelper is newer and used more, TestUtils#createResource 
> should be replaced with ResourceTypesTestHelper#newResource with all 
> occurrence.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9997) Code cleanup in ZKConfigurationStore

2020-03-09 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9997:
---
Attachment: YARN-9997.004.patch

> Code cleanup in ZKConfigurationStore
> 
>
> Key: YARN-9997
> URL: https://issues.apache.org/jira/browse/YARN-9997
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-9997.001.patch, YARN-9997.002.patch, 
> YARN-9997.003.patch, YARN-9997.004.patch
>
>
> Many thins can be improved:
> * znodeParentPath could be a local variable
> * zkManager could be private, VisibleForTesting annotation is not needed 
> anymore
> * Do something with unchecked casts
> * zkManager.safeSetData calls are almost having the same set of parameters: 
> Simplify this
> * Extract zkManager calls to their own methods: They are repeated
> * Remove TODOs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9354) TestUtils#createResource calls should be replaced with ResourceTypesTestHelper#newResource

2020-03-09 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9354:
---
Attachment: YARN-9354.004.patch

> TestUtils#createResource calls should be replaced with 
> ResourceTypesTestHelper#newResource
> --
>
> Key: YARN-9354
> URL: https://issues.apache.org/jira/browse/YARN-9354
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Trivial
>  Labels: newbie, newbie++
> Attachments: YARN-9354.001.patch, YARN-9354.002.patch, 
> YARN-9354.003.patch, YARN-9354.004.patch
>
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestUtils#createResource
>  has not identical, but very similar implementation to 
> org.apache.hadoop.yarn.resourcetypes.ResourceTypesTestHelper#newResource. 
> Since these 2 methods are doing the same essentially and 
> ResourceTypesTestHelper is newer and used more, TestUtils#createResource 
> should be replaced with ResourceTypesTestHelper#newResource with all 
> occurrence.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9997) Code cleanup in ZKConfigurationStore

2020-03-09 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9997:
---
Attachment: YARN-9997.002.patch

> Code cleanup in ZKConfigurationStore
> 
>
> Key: YARN-9997
> URL: https://issues.apache.org/jira/browse/YARN-9997
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-9997.001.patch, YARN-9997.002.patch
>
>
> Many thins can be improved:
> * znodeParentPath could be a local variable
> * zkManager could be private, VisibleForTesting annotation is not needed 
> anymore
> * Do something with unchecked casts
> * zkManager.safeSetData calls are almost having the same set of parameters: 
> Simplify this
> * Extract zkManager calls to their own methods: They are repeated
> * Remove TODOs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9997) Code cleanup in ZKConfigurationStore

2020-03-09 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054919#comment-17054919
 ] 

Andras Gyori commented on YARN-9997:


Thank you for feedback, [~snemeth].

I have addressed all the issues, and reinforced them with additional unit 
tests. #logMutation should not overwrite the data when an invalid (non 
linked-list) object is queried from Zookeeper logpath. As for the third 
concern, receiving a null value as a config version is considered to be an 
invalid state and an IllegalStateException is thrown accordingly.

> Code cleanup in ZKConfigurationStore
> 
>
> Key: YARN-9997
> URL: https://issues.apache.org/jira/browse/YARN-9997
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-9997.001.patch, YARN-9997.002.patch
>
>
> Many thins can be improved:
> * znodeParentPath could be a local variable
> * zkManager could be private, VisibleForTesting annotation is not needed 
> anymore
> * Do something with unchecked casts
> * zkManager.safeSetData calls are almost having the same set of parameters: 
> Simplify this
> * Extract zkManager calls to their own methods: They are repeated
> * Remove TODOs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9354) Resources should be created with ResourceTypesTestHelper instead of TestUtils

2020-03-13 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9354:
---
Attachment: YARN-9354.branch-3.2.003.patch

> Resources should be created with ResourceTypesTestHelper instead of TestUtils
> -
>
> Key: YARN-9354
> URL: https://issues.apache.org/jira/browse/YARN-9354
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Trivial
>  Labels: newbie, newbie++
> Fix For: 3.3.0
>
> Attachments: YARN-9354.001.patch, YARN-9354.002.patch, 
> YARN-9354.003.patch, YARN-9354.004.patch, YARN-9354.branch-3.2.001.patch, 
> YARN-9354.branch-3.2.002.patch, YARN-9354.branch-3.2.003.patch
>
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestUtils#createResource
>  has not identical, but very similar implementation to 
> org.apache.hadoop.yarn.resourcetypes.ResourceTypesTestHelper#newResource. 
> Since these 2 methods are doing the same essentially and 
> ResourceTypesTestHelper is newer and used more, TestUtils#createResource 
> should be replaced with ResourceTypesTestHelper#newResource with all 
> occurrence.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9354) Resources should be created with ResourceTypesTestHelper instead of TestUtils

2020-03-11 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9354:
---
Attachment: YARN-9354.branch-3.2.002.patch

> Resources should be created with ResourceTypesTestHelper instead of TestUtils
> -
>
> Key: YARN-9354
> URL: https://issues.apache.org/jira/browse/YARN-9354
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Trivial
>  Labels: newbie, newbie++
> Fix For: 3.3.0
>
> Attachments: YARN-9354.001.patch, YARN-9354.002.patch, 
> YARN-9354.003.patch, YARN-9354.004.patch, YARN-9354.branch-3.2.001.patch, 
> YARN-9354.branch-3.2.002.patch
>
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestUtils#createResource
>  has not identical, but very similar implementation to 
> org.apache.hadoop.yarn.resourcetypes.ResourceTypesTestHelper#newResource. 
> Since these 2 methods are doing the same essentially and 
> ResourceTypesTestHelper is newer and used more, TestUtils#createResource 
> should be replaced with ResourceTypesTestHelper#newResource with all 
> occurrence.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-10236) LogAggregation cycles might collide with log writes

2020-04-16 Thread Andras Gyori (Jira)
Andras Gyori created YARN-10236:
---

 Summary: LogAggregation cycles might collide with log writes
 Key: YARN-10236
 URL: https://issues.apache.org/jira/browse/YARN-10236
 Project: Hadoop YARN
  Issue Type: Bug
  Components: log-aggregation
Affects Versions: 3.3.0
Reporter: Andras Gyori


A LogAggregation cycle might interrupt an in-progress logging. Chance of this 
is happening is low, however, not zero. Due to this race condition, a log entry 
could be entirely discarded, without being written to the log file.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10236) LogAggregation cycles might collide with log writes

2020-04-16 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10236:

 Attachment: YARN-10236.001.log
Description: 
A LogAggregation cycle might interrupt an in-progress logging. Chance of this 
is happening is low, however, not zero. Due to this race condition, a log entry 
could be entirely discarded, without being written to the log file.

Tested with specifications:
 * Kerberos + SSL enabled
 * TFile + Rolling log aggregation

The log file misses the 58th entry due to this behaviour.

  was:A LogAggregation cycle might interrupt an in-progress logging. Chance of 
this is happening is low, however, not zero. Due to this race condition, a log 
entry could be entirely discarded, without being written to the log file.


> LogAggregation cycles might collide with log writes
> ---
>
> Key: YARN-10236
> URL: https://issues.apache.org/jira/browse/YARN-10236
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: log-aggregation
>Affects Versions: 3.3.0
>Reporter: Andras Gyori
>Priority: Major
> Attachments: YARN-10236.001.log
>
>
> A LogAggregation cycle might interrupt an in-progress logging. Chance of this 
> is happening is low, however, not zero. Due to this race condition, a log 
> entry could be entirely discarded, without being written to the log file.
> Tested with specifications:
>  * Kerberos + SSL enabled
>  * TFile + Rolling log aggregation
> The log file misses the 58th entry due to this behaviour.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser

2020-03-16 Thread Andras Gyori (Jira)
Andras Gyori created YARN-10199:
---

 Summary: Simplify UserGroupMappingPlacementRule#getPlacementForUser
 Key: YARN-10199
 URL: https://issues.apache.org/jira/browse/YARN-10199
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: scheduler
Reporter: Andras Gyori
Assignee: Andras Gyori


The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly 
responsible for queue naming, contains deeply nested branches. In order to 
provide an extendable mapping logic, the branches could be flattened and 
simplified.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10215) Endpoint for obtaining direct URL for the logs

2020-04-01 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10215:

Attachment: YARN-10025.002.patch

> Endpoint for obtaining direct URL for the logs
> --
>
> Key: YARN-10215
> URL: https://issues.apache.org/jira/browse/YARN-10215
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10025.001.patch, YARN-10025.002.patch
>
>
> If CORS protected UIs are set up, there is an issue when the browser tries to 
> access the logs of a running container in the RM web UIv2.
> Assuming ATS is not up, the browser follows the following call chain:
> - Tries to access ATS, it fails, falls back to JHS
> - From RM the browser received basic app info, we know that the application 
> is running
> - From the JHS we got the list of containers and their log files.
> - When we try to access a specific log file, the JHS redirects the request to 
> the NM's UI (on which node the container is running). This redirect is 
> performed by the browser automatically. In this setup the host is considered 
> as a protected information, thus the browser omits the "Origin" field from 
> the request when this redirect is done. The browser then denies access to the 
> NodeManager's web UI due to the CORS header set up for NM, but the Origin is 
> null in the redirect request. 
> - Finally, "Logs are unavailable" message is shown in the RM web UIv2 due to 
> the CORS violation.
> We should fix this. As an approach we can expose another endpoints which only 
> returns the URL of the NodeManager what we should call directly from the UIv2 
> in order to receive the log. This adds a bit of a complexity, but will enable 
> users to keep the CORS protected setup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10215) Endpoint for obtaining direct URL for the logs

2020-04-06 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076083#comment-17076083
 ] 

Andras Gyori commented on YARN-10215:
-

Thanks [~adam.antal] for the fast review.
The checkstyle issues have been resolved apart from a too many arguments issue, 
for which I am afraid this patch is out of scope. The findbug issue was not 
introduced by this patch, hence I would keep it as is. The unnecessary log 
entries were removed. 

> Endpoint for obtaining direct URL for the logs
> --
>
> Key: YARN-10215
> URL: https://issues.apache.org/jira/browse/YARN-10215
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10025.001.patch, YARN-10025.002.patch, 
> YARN-10025.003.patch
>
>
> If CORS protected UIs are set up, there is an issue when the browser tries to 
> access the logs of a running container in the RM web UIv2.
> Assuming ATS is not up, the browser follows the following call chain:
> - Tries to access ATS, it fails, falls back to JHS
> - From RM the browser received basic app info, we know that the application 
> is running
> - From the JHS we got the list of containers and their log files.
> - When we try to access a specific log file, the JHS redirects the request to 
> the NM's UI (on which node the container is running). This redirect is 
> performed by the browser automatically. In this setup the host is considered 
> as a protected information, thus the browser omits the "Origin" field from 
> the request when this redirect is done. The browser then denies access to the 
> NodeManager's web UI due to the CORS header set up for NM, but the Origin is 
> null in the redirect request. 
> - Finally, "Logs are unavailable" message is shown in the RM web UIv2 due to 
> the CORS violation.
> We should fix this. As an approach we can expose another endpoints which only 
> returns the URL of the NodeManager what we should call directly from the UIv2 
> in order to receive the log. This adds a bit of a complexity, but will enable 
> users to keep the CORS protected setup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10215) Endpoint for obtaining direct URL for the logs

2020-04-03 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10215:

Attachment: YARN-10025.003.patch

> Endpoint for obtaining direct URL for the logs
> --
>
> Key: YARN-10215
> URL: https://issues.apache.org/jira/browse/YARN-10215
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10025.001.patch, YARN-10025.002.patch, 
> YARN-10025.003.patch
>
>
> If CORS protected UIs are set up, there is an issue when the browser tries to 
> access the logs of a running container in the RM web UIv2.
> Assuming ATS is not up, the browser follows the following call chain:
> - Tries to access ATS, it fails, falls back to JHS
> - From RM the browser received basic app info, we know that the application 
> is running
> - From the JHS we got the list of containers and their log files.
> - When we try to access a specific log file, the JHS redirects the request to 
> the NM's UI (on which node the container is running). This redirect is 
> performed by the browser automatically. In this setup the host is considered 
> as a protected information, thus the browser omits the "Origin" field from 
> the request when this redirect is done. The browser then denies access to the 
> NodeManager's web UI due to the CORS header set up for NM, but the Origin is 
> null in the redirect request. 
> - Finally, "Logs are unavailable" message is shown in the RM web UIv2 due to 
> the CORS violation.
> We should fix this. As an approach we can expose another endpoints which only 
> returns the URL of the NodeManager what we should call directly from the UIv2 
> in order to receive the log. This adds a bit of a complexity, but will enable 
> users to keep the CORS protected setup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser

2020-03-30 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070811#comment-17070811
 ] 

Andras Gyori commented on YARN-10199:
-

Thank you [~pbacsko] for the feedback. I have applied you advises as detailed 
below:
 # stored the frequently used values in a local variable
 # Removed the redundant GROUP_MAPPING checks, and simplified the boolean 
expression
 # 4. Fixed 

As for now, I hold the same position as you. Further granulation of this logic 
is not justified yet, however, it will be easier to migrate to a class-based 
approach from this state.

> Simplify UserGroupMappingPlacementRule#getPlacementForUser
> --
>
> Key: YARN-10199
> URL: https://issues.apache.org/jira/browse/YARN-10199
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10199.001.patch, YARN-10199.002.patch, 
> YARN-10199.003.patch, YARN-10199.004.patch, YARN-10199.005.patch
>
>
> The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly 
> responsible for queue naming, contains deeply nested branches. In order to 
> provide an extendable mapping logic, the branches could be flattened and 
> simplified.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser

2020-03-30 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070811#comment-17070811
 ] 

Andras Gyori edited comment on YARN-10199 at 3/30/20, 9:14 AM:
---

Thank you [~pbacsko] for the feedback. I have applied your recommendations as 
detailed below:
 # stored the frequently used values in a local variable
 # removed the redundant GROUP_MAPPING checks, and simplified the boolean 
expression
 # 4. Fixed 

As for now, I hold the same position as you. Further granulation of this logic 
is not justified yet, however, it will be easier to migrate to a class-based 
approach from this state.


was (Author: gandras):
Thank you [~pbacsko] for the feedback. I have applied you advises as detailed 
below:
 # stored the frequently used values in a local variable
 # Removed the redundant GROUP_MAPPING checks, and simplified the boolean 
expression
 # 4. Fixed 

As for now, I hold the same position as you. Further granulation of this logic 
is not justified yet, however, it will be easier to migrate to a class-based 
approach from this state.

> Simplify UserGroupMappingPlacementRule#getPlacementForUser
> --
>
> Key: YARN-10199
> URL: https://issues.apache.org/jira/browse/YARN-10199
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10199.001.patch, YARN-10199.002.patch, 
> YARN-10199.003.patch, YARN-10199.004.patch, YARN-10199.005.patch
>
>
> The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly 
> responsible for queue naming, contains deeply nested branches. In order to 
> provide an extendable mapping logic, the branches could be flattened and 
> simplified.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser

2020-03-30 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10199:

Attachment: YARN-10199.005.patch

> Simplify UserGroupMappingPlacementRule#getPlacementForUser
> --
>
> Key: YARN-10199
> URL: https://issues.apache.org/jira/browse/YARN-10199
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10199.001.patch, YARN-10199.002.patch, 
> YARN-10199.003.patch, YARN-10199.004.patch, YARN-10199.005.patch
>
>
> The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly 
> responsible for queue naming, contains deeply nested branches. In order to 
> provide an extendable mapping logic, the branches could be flattened and 
> simplified.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10215) Endpoint for obtaining direct URL for the logs

2020-04-01 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072898#comment-17072898
 ] 

Andras Gyori commented on YARN-10215:
-

Thank you [~adam.antal] for the review. I have addressed your concerns with the 
following additions:
 # A manual_redirection query parameter has been added to the corresponding 
endpoints. It is false by default, and if turned on, the automatic redirect (a 
307 response) is swapped with a 206 response, that could be used to handle the 
redirection manually (as the UI does it).
 # I think a null check is already in place in the handleResponse function.
 # The createEmptyContainerLogInfo has been extracted to a helper file for 
common use.

 

> Endpoint for obtaining direct URL for the logs
> --
>
> Key: YARN-10215
> URL: https://issues.apache.org/jira/browse/YARN-10215
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10025.001.patch, YARN-10025.002.patch
>
>
> If CORS protected UIs are set up, there is an issue when the browser tries to 
> access the logs of a running container in the RM web UIv2.
> Assuming ATS is not up, the browser follows the following call chain:
> - Tries to access ATS, it fails, falls back to JHS
> - From RM the browser received basic app info, we know that the application 
> is running
> - From the JHS we got the list of containers and their log files.
> - When we try to access a specific log file, the JHS redirects the request to 
> the NM's UI (on which node the container is running). This redirect is 
> performed by the browser automatically. In this setup the host is considered 
> as a protected information, thus the browser omits the "Origin" field from 
> the request when this redirect is done. The browser then denies access to the 
> NodeManager's web UI due to the CORS header set up for NM, but the Origin is 
> null in the redirect request. 
> - Finally, "Logs are unavailable" message is shown in the RM web UIv2 due to 
> the CORS violation.
> We should fix this. As an approach we can expose another endpoints which only 
> returns the URL of the NodeManager what we should call directly from the UIv2 
> in order to receive the log. This adds a bit of a complexity, but will enable 
> users to keep the CORS protected setup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser

2020-03-26 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10199:

Attachment: YARN-10199.004.patch

> Simplify UserGroupMappingPlacementRule#getPlacementForUser
> --
>
> Key: YARN-10199
> URL: https://issues.apache.org/jira/browse/YARN-10199
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10199.001.patch, YARN-10199.002.patch, 
> YARN-10199.003.patch, YARN-10199.004.patch
>
>
> The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly 
> responsible for queue naming, contains deeply nested branches. In order to 
> provide an extendable mapping logic, the branches could be flattened and 
> simplified.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser

2020-03-26 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067612#comment-17067612
 ] 

Andras Gyori edited comment on YARN-10199 at 3/26/20, 11:41 AM:


The new version takes into account the changes which were introduced in 
YARN-10198 and YARN-9879.


was (Author: gandras):
The new version takes into account the changes that was introduced in 
[YARN-10198|https://issues.apache.org/jira/browse/YARN-10198] and 
[YARN-9879|https://issues.apache.org/jira/browse/YARN-9879].

> Simplify UserGroupMappingPlacementRule#getPlacementForUser
> --
>
> Key: YARN-10199
> URL: https://issues.apache.org/jira/browse/YARN-10199
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10199.001.patch, YARN-10199.002.patch, 
> YARN-10199.003.patch, YARN-10199.004.patch
>
>
> The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly 
> responsible for queue naming, contains deeply nested branches. In order to 
> provide an extendable mapping logic, the branches could be flattened and 
> simplified.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser

2020-03-26 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067612#comment-17067612
 ] 

Andras Gyori commented on YARN-10199:
-

The new version takes into account the changes that was introduced in 
[YARN-10198|https://issues.apache.org/jira/browse/YARN-10198] and 
[YARN-9879|https://issues.apache.org/jira/browse/YARN-9879].

> Simplify UserGroupMappingPlacementRule#getPlacementForUser
> --
>
> Key: YARN-10199
> URL: https://issues.apache.org/jira/browse/YARN-10199
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10199.001.patch, YARN-10199.002.patch, 
> YARN-10199.003.patch, YARN-10199.004.patch
>
>
> The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly 
> responsible for queue naming, contains deeply nested branches. In order to 
> provide an extendable mapping logic, the branches could be flattened and 
> simplified.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9354) Resources should be created with ResourceTypesTestHelper instead of TestUtils

2020-03-25 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066468#comment-17066468
 ] 

Andras Gyori commented on YARN-9354:


A new patch has been submitted for the branch-3.2 backport, the failing unit 
tests are unrelated.

> Resources should be created with ResourceTypesTestHelper instead of TestUtils
> -
>
> Key: YARN-9354
> URL: https://issues.apache.org/jira/browse/YARN-9354
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Trivial
>  Labels: newbie, newbie++
> Fix For: 3.3.0
>
> Attachments: YARN-9354.001.patch, YARN-9354.002.patch, 
> YARN-9354.003.patch, YARN-9354.004.patch, YARN-9354.branch-3.2.001.patch, 
> YARN-9354.branch-3.2.002.patch, YARN-9354.branch-3.2.003.patch
>
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestUtils#createResource
>  has not identical, but very similar implementation to 
> org.apache.hadoop.yarn.resourcetypes.ResourceTypesTestHelper#newResource. 
> Since these 2 methods are doing the same essentially and 
> ResourceTypesTestHelper is newer and used more, TestUtils#createResource 
> should be replaced with ResourceTypesTestHelper#newResource with all 
> occurrence.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9997) Code cleanup in ZKConfigurationStore

2020-03-25 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066471#comment-17066471
 ] 

Andras Gyori commented on YARN-9997:


The backport is ready to be merged, however, I am waiting for the update of 
[YARN-10002|https://issues.apache.org/jira/browse/YARN-10002], whether that is 
possible to be backported as well. In that case, YARN-10002 should be merged 
first to avoid conflicts.

> Code cleanup in ZKConfigurationStore
> 
>
> Key: YARN-9997
> URL: https://issues.apache.org/jira/browse/YARN-9997
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9997.001.patch, YARN-9997.002.patch, 
> YARN-9997.003.patch, YARN-9997.004.patch, YARN-9997.005.patch, 
> YARN-9997.006.patch
>
>
> Many thins can be improved:
> * znodeParentPath could be a local variable
> * zkManager could be private, VisibleForTesting annotation is not needed 
> anymore
> * Do something with unchecked casts
> * zkManager.safeSetData calls are almost having the same set of parameters: 
> Simplify this
> * Extract zkManager calls to their own methods: They are repeated
> * Remove TODOs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9997) Code cleanup in ZKConfigurationStore

2020-04-22 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9997:
---
Attachment: YARN-9997.branch-3.2.001.patch

> Code cleanup in ZKConfigurationStore
> 
>
> Key: YARN-9997
> URL: https://issues.apache.org/jira/browse/YARN-9997
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9997.001.patch, YARN-9997.002.patch, 
> YARN-9997.003.patch, YARN-9997.004.patch, YARN-9997.005.patch, 
> YARN-9997.006.patch, YARN-9997.branch-3.2.001.patch
>
>
> Many thins can be improved:
> * znodeParentPath could be a local variable
> * zkManager could be private, VisibleForTesting annotation is not needed 
> anymore
> * Do something with unchecked casts
> * zkManager.safeSetData calls are almost having the same set of parameters: 
> Simplify this
> * Extract zkManager calls to their own methods: They are repeated
> * Remove TODOs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9997) Code cleanup in ZKConfigurationStore

2020-04-22 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089744#comment-17089744
 ] 

Andras Gyori commented on YARN-9997:


As YARN-10002 had been successfully backported, I have uploaded the backport 
patch as well.

> Code cleanup in ZKConfigurationStore
> 
>
> Key: YARN-9997
> URL: https://issues.apache.org/jira/browse/YARN-9997
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9997.001.patch, YARN-9997.002.patch, 
> YARN-9997.003.patch, YARN-9997.004.patch, YARN-9997.005.patch, 
> YARN-9997.006.patch, YARN-9997.branch-3.2.001.patch
>
>
> Many thins can be improved:
> * znodeParentPath could be a local variable
> * zkManager could be private, VisibleForTesting annotation is not needed 
> anymore
> * Do something with unchecked casts
> * zkManager.safeSetData calls are almost having the same set of parameters: 
> Simplify this
> * Extract zkManager calls to their own methods: They are repeated
> * Remove TODOs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10215) Endpoint for obtaining direct URL for the logs

2020-04-28 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10215:

Attachment: YARN-10025.004.patch

> Endpoint for obtaining direct URL for the logs
> --
>
> Key: YARN-10215
> URL: https://issues.apache.org/jira/browse/YARN-10215
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10025.001.patch, YARN-10025.002.patch, 
> YARN-10025.003.patch, YARN-10025.004.patch
>
>
> If CORS protected UIs are set up, there is an issue when the browser tries to 
> access the logs of a running container in the RM web UIv2.
> Assuming ATS is not up, the browser follows the following call chain:
> - Tries to access ATS, it fails, falls back to JHS
> - From RM the browser received basic app info, we know that the application 
> is running
> - From the JHS we got the list of containers and their log files.
> - When we try to access a specific log file, the JHS redirects the request to 
> the NM's UI (on which node the container is running). This redirect is 
> performed by the browser automatically. In this setup the host is considered 
> as a protected information, thus the browser omits the "Origin" field from 
> the request when this redirect is done. The browser then denies access to the 
> NodeManager's web UI due to the CORS header set up for NM, but the Origin is 
> null in the redirect request. 
> - Finally, "Logs are unavailable" message is shown in the RM web UIv2 due to 
> the CORS violation.
> We should fix this. As an approach we can expose another endpoints which only 
> returns the URL of the NodeManager what we should call directly from the UIv2 
> in order to receive the log. This adds a bit of a complexity, but will enable 
> users to keep the CORS protected setup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10215) Endpoint for obtaining direct URL for the logs

2020-04-28 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094242#comment-17094242
 ] 

Andras Gyori edited comment on YARN-10215 at 4/28/20, 8:03 AM:
---

Updated the patch to return the redirected url in a 200 response rather than in 
a 206 response. 


was (Author: gandras):
Updated the patch to return the redirected url in a 202 response rather than in 
a 206 response. 

> Endpoint for obtaining direct URL for the logs
> --
>
> Key: YARN-10215
> URL: https://issues.apache.org/jira/browse/YARN-10215
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10025.001.patch, YARN-10025.002.patch, 
> YARN-10025.003.patch, YARN-10025.004.patch
>
>
> If CORS protected UIs are set up, there is an issue when the browser tries to 
> access the logs of a running container in the RM web UIv2.
> Assuming ATS is not up, the browser follows the following call chain:
> - Tries to access ATS, it fails, falls back to JHS
> - From RM the browser received basic app info, we know that the application 
> is running
> - From the JHS we got the list of containers and their log files.
> - When we try to access a specific log file, the JHS redirects the request to 
> the NM's UI (on which node the container is running). This redirect is 
> performed by the browser automatically. In this setup the host is considered 
> as a protected information, thus the browser omits the "Origin" field from 
> the request when this redirect is done. The browser then denies access to the 
> NodeManager's web UI due to the CORS header set up for NM, but the Origin is 
> null in the redirect request. 
> - Finally, "Logs are unavailable" message is shown in the RM web UIv2 due to 
> the CORS violation.
> We should fix this. As an approach we can expose another endpoints which only 
> returns the URL of the NodeManager what we should call directly from the UIv2 
> in order to receive the log. This adds a bit of a complexity, but will enable 
> users to keep the CORS protected setup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10215) Endpoint for obtaining direct URL for the logs

2020-04-28 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10215:

Attachment: (was: YARN-10025.004.patch)

> Endpoint for obtaining direct URL for the logs
> --
>
> Key: YARN-10215
> URL: https://issues.apache.org/jira/browse/YARN-10215
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10025.001.patch, YARN-10025.002.patch, 
> YARN-10025.003.patch, YARN-10025.004.patch
>
>
> If CORS protected UIs are set up, there is an issue when the browser tries to 
> access the logs of a running container in the RM web UIv2.
> Assuming ATS is not up, the browser follows the following call chain:
> - Tries to access ATS, it fails, falls back to JHS
> - From RM the browser received basic app info, we know that the application 
> is running
> - From the JHS we got the list of containers and their log files.
> - When we try to access a specific log file, the JHS redirects the request to 
> the NM's UI (on which node the container is running). This redirect is 
> performed by the browser automatically. In this setup the host is considered 
> as a protected information, thus the browser omits the "Origin" field from 
> the request when this redirect is done. The browser then denies access to the 
> NodeManager's web UI due to the CORS header set up for NM, but the Origin is 
> null in the redirect request. 
> - Finally, "Logs are unavailable" message is shown in the RM web UIv2 due to 
> the CORS violation.
> We should fix this. As an approach we can expose another endpoints which only 
> returns the URL of the NodeManager what we should call directly from the UIv2 
> in order to receive the log. This adds a bit of a complexity, but will enable 
> users to keep the CORS protected setup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10215) Endpoint for obtaining direct URL for the logs

2020-04-28 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10215:

Attachment: YARN-10025.004.patch

> Endpoint for obtaining direct URL for the logs
> --
>
> Key: YARN-10215
> URL: https://issues.apache.org/jira/browse/YARN-10215
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10025.001.patch, YARN-10025.002.patch, 
> YARN-10025.003.patch, YARN-10025.004.patch
>
>
> If CORS protected UIs are set up, there is an issue when the browser tries to 
> access the logs of a running container in the RM web UIv2.
> Assuming ATS is not up, the browser follows the following call chain:
> - Tries to access ATS, it fails, falls back to JHS
> - From RM the browser received basic app info, we know that the application 
> is running
> - From the JHS we got the list of containers and their log files.
> - When we try to access a specific log file, the JHS redirects the request to 
> the NM's UI (on which node the container is running). This redirect is 
> performed by the browser automatically. In this setup the host is considered 
> as a protected information, thus the browser omits the "Origin" field from 
> the request when this redirect is done. The browser then denies access to the 
> NodeManager's web UI due to the CORS header set up for NM, but the Origin is 
> null in the redirect request. 
> - Finally, "Logs are unavailable" message is shown in the RM web UIv2 due to 
> the CORS violation.
> We should fix this. As an approach we can expose another endpoints which only 
> returns the URL of the NodeManager what we should call directly from the UIv2 
> in order to receive the log. This adds a bit of a complexity, but will enable 
> users to keep the CORS protected setup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10215) Endpoint for obtaining direct URL for the logs

2020-04-28 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094242#comment-17094242
 ] 

Andras Gyori commented on YARN-10215:
-

Updated the patch to return the redirected url in a 202 response rather than a 
206. response. 

> Endpoint for obtaining direct URL for the logs
> --
>
> Key: YARN-10215
> URL: https://issues.apache.org/jira/browse/YARN-10215
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10025.001.patch, YARN-10025.002.patch, 
> YARN-10025.003.patch, YARN-10025.004.patch
>
>
> If CORS protected UIs are set up, there is an issue when the browser tries to 
> access the logs of a running container in the RM web UIv2.
> Assuming ATS is not up, the browser follows the following call chain:
> - Tries to access ATS, it fails, falls back to JHS
> - From RM the browser received basic app info, we know that the application 
> is running
> - From the JHS we got the list of containers and their log files.
> - When we try to access a specific log file, the JHS redirects the request to 
> the NM's UI (on which node the container is running). This redirect is 
> performed by the browser automatically. In this setup the host is considered 
> as a protected information, thus the browser omits the "Origin" field from 
> the request when this redirect is done. The browser then denies access to the 
> NodeManager's web UI due to the CORS header set up for NM, but the Origin is 
> null in the redirect request. 
> - Finally, "Logs are unavailable" message is shown in the RM web UIv2 due to 
> the CORS violation.
> We should fix this. As an approach we can expose another endpoints which only 
> returns the URL of the NodeManager what we should call directly from the UIv2 
> in order to receive the log. This adds a bit of a complexity, but will enable 
> users to keep the CORS protected setup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10215) Endpoint for obtaining direct URL for the logs

2020-04-28 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094242#comment-17094242
 ] 

Andras Gyori edited comment on YARN-10215 at 4/28/20, 7:35 AM:
---

Updated the patch to return the redirected url in a 202 response rather than in 
a 206 response. 


was (Author: gandras):
Updated the patch to return the redirected url in a 202 response rather than a 
206. response. 

> Endpoint for obtaining direct URL for the logs
> --
>
> Key: YARN-10215
> URL: https://issues.apache.org/jira/browse/YARN-10215
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10025.001.patch, YARN-10025.002.patch, 
> YARN-10025.003.patch, YARN-10025.004.patch
>
>
> If CORS protected UIs are set up, there is an issue when the browser tries to 
> access the logs of a running container in the RM web UIv2.
> Assuming ATS is not up, the browser follows the following call chain:
> - Tries to access ATS, it fails, falls back to JHS
> - From RM the browser received basic app info, we know that the application 
> is running
> - From the JHS we got the list of containers and their log files.
> - When we try to access a specific log file, the JHS redirects the request to 
> the NM's UI (on which node the container is running). This redirect is 
> performed by the browser automatically. In this setup the host is considered 
> as a protected information, thus the browser omits the "Origin" field from 
> the request when this redirect is done. The browser then denies access to the 
> NodeManager's web UI due to the CORS header set up for NM, but the Origin is 
> null in the redirect request. 
> - Finally, "Logs are unavailable" message is shown in the RM web UIv2 due to 
> the CORS violation.
> We should fix this. As an approach we can expose another endpoints which only 
> returns the URL of the NodeManager what we should call directly from the UIv2 
> in order to receive the log. This adds a bit of a complexity, but will enable 
> users to keep the CORS protected setup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser

2020-04-29 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095223#comment-17095223
 ] 

Andras Gyori commented on YARN-10199:
-

Uploaded a new revision which is rebased on trunk, containing the change 
introduced in YARN-10226.

> Simplify UserGroupMappingPlacementRule#getPlacementForUser
> --
>
> Key: YARN-10199
> URL: https://issues.apache.org/jira/browse/YARN-10199
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10199.001.patch, YARN-10199.002.patch, 
> YARN-10199.003.patch, YARN-10199.004.patch, YARN-10199.005.patch, 
> YARN-10199.006.patch
>
>
> The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly 
> responsible for queue naming, contains deeply nested branches. In order to 
> provide an extendable mapping logic, the branches could be flattened and 
> simplified.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser

2020-04-29 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10199:

Attachment: YARN-10199.006.patch

> Simplify UserGroupMappingPlacementRule#getPlacementForUser
> --
>
> Key: YARN-10199
> URL: https://issues.apache.org/jira/browse/YARN-10199
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10199.001.patch, YARN-10199.002.patch, 
> YARN-10199.003.patch, YARN-10199.004.patch, YARN-10199.005.patch, 
> YARN-10199.006.patch
>
>
> The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly 
> responsible for queue naming, contains deeply nested branches. In order to 
> provide an extendable mapping logic, the branches could be flattened and 
> simplified.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser

2020-04-29 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10199:

Attachment: YARN-10199.007.patch

> Simplify UserGroupMappingPlacementRule#getPlacementForUser
> --
>
> Key: YARN-10199
> URL: https://issues.apache.org/jira/browse/YARN-10199
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10199.001.patch, YARN-10199.002.patch, 
> YARN-10199.003.patch, YARN-10199.004.patch, YARN-10199.005.patch, 
> YARN-10199.006.patch, YARN-10199.007.patch
>
>
> The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly 
> responsible for queue naming, contains deeply nested branches. In order to 
> provide an extendable mapping logic, the branches could be flattened and 
> simplified.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10179) Queue mapping based on group id passed through application tag

2020-03-19 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10179:

Attachment: YARN-10179.001.patch

> Queue mapping based on group id passed through application tag
> --
>
> Key: YARN-10179
> URL: https://issues.apache.org/jira/browse/YARN-10179
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-10179.001.patch
>
>
> There are situations when the real submitting user differs from the user what 
> arrives to YARN. For example in case of a Hive application when Hive 
> impersonation is turned off, the hive queries will run as Hive user and the 
> mapping is done based on the user's group. 
> Unfortunately in this case YARN doesn't have any information about the real 
> user and there are cases when the customer may want to map these applications 
> to the real submitting user's queue (based on the group id) instead of the 
> Hive queue.
> For these cases, if they would pass the group id (or name) in the application 
> tag we may read it and use it during the queue mapping, if that user has 
> rights to run on the real user's queue.  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10179) Queue mapping based on group id passed through application tag

2020-03-19 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10179:

Attachment: (was: YARN-10179.001.patch)

> Queue mapping based on group id passed through application tag
> --
>
> Key: YARN-10179
> URL: https://issues.apache.org/jira/browse/YARN-10179
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Major
> Fix For: 3.3.0
>
>
> There are situations when the real submitting user differs from the user what 
> arrives to YARN. For example in case of a Hive application when Hive 
> impersonation is turned off, the hive queries will run as Hive user and the 
> mapping is done based on the user's group. 
> Unfortunately in this case YARN doesn't have any information about the real 
> user and there are cases when the customer may want to map these applications 
> to the real submitting user's queue (based on the group id) instead of the 
> Hive queue.
> For these cases, if they would pass the group id (or name) in the application 
> tag we may read it and use it during the queue mapping, if that user has 
> rights to run on the real user's queue.  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser

2020-03-19 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10199:

Attachment: YARN-10199.003.patch

> Simplify UserGroupMappingPlacementRule#getPlacementForUser
> --
>
> Key: YARN-10199
> URL: https://issues.apache.org/jira/browse/YARN-10199
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10199.001.patch, YARN-10199.002.patch, 
> YARN-10199.003.patch
>
>
> The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly 
> responsible for queue naming, contains deeply nested branches. In order to 
> provide an extendable mapping logic, the branches could be flattened and 
> simplified.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10179) Queue mapping based on group id passed through application tag

2020-03-19 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10179:

Attachment: YARN-10179.002.patch

> Queue mapping based on group id passed through application tag
> --
>
> Key: YARN-10179
> URL: https://issues.apache.org/jira/browse/YARN-10179
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-10179.001.patch, YARN-10179.002.patch
>
>
> There are situations when the real submitting user differs from the user what 
> arrives to YARN. For example in case of a Hive application when Hive 
> impersonation is turned off, the hive queries will run as Hive user and the 
> mapping is done based on the user's group. 
> Unfortunately in this case YARN doesn't have any information about the real 
> user and there are cases when the customer may want to map these applications 
> to the real submitting user's queue (based on the group id) instead of the 
> Hive queue.
> For these cases, if they would pass the group id (or name) in the application 
> tag we may read it and use it during the queue mapping, if that user has 
> rights to run on the real user's queue.  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser

2020-03-17 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10199:

Attachment: YARN-10199.002.patch

> Simplify UserGroupMappingPlacementRule#getPlacementForUser
> --
>
> Key: YARN-10199
> URL: https://issues.apache.org/jira/browse/YARN-10199
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10199.001.patch, YARN-10199.002.patch
>
>
> The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly 
> responsible for queue naming, contains deeply nested branches. In order to 
> provide an extendable mapping logic, the branches could be flattened and 
> simplified.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser

2020-03-17 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061032#comment-17061032
 ] 

Andras Gyori commented on YARN-10199:
-

Thank you  for the suggestion [~pbacsko].

It is indeed a very good idea and would facilitate the testing of queue 
mapping. I have acted accordingly and the logic, with the involved methods have 
been extracted to their own class.

> Simplify UserGroupMappingPlacementRule#getPlacementForUser
> --
>
> Key: YARN-10199
> URL: https://issues.apache.org/jira/browse/YARN-10199
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10199.001.patch
>
>
> The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly 
> responsible for queue naming, contains deeply nested branches. In order to 
> provide an extendable mapping logic, the branches could be flattened and 
> simplified.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser

2020-03-17 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10199:

Attachment: YARN-10199.001.patch

> Simplify UserGroupMappingPlacementRule#getPlacementForUser
> --
>
> Key: YARN-10199
> URL: https://issues.apache.org/jira/browse/YARN-10199
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10199.001.patch
>
>
> The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly 
> responsible for queue naming, contains deeply nested branches. In order to 
> provide an extendable mapping logic, the branches could be flattened and 
> simplified.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9419) Log a warning if GPU isolation is enabled but LinuxContainerExecutor is disabled

2020-03-09 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-9419:
---
Attachment: YARN-9419.branch-3.2.001.patch

> Log a warning if GPU isolation is enabled but LinuxContainerExecutor is 
> disabled
> 
>
> Key: YARN-9419
> URL: https://issues.apache.org/jira/browse/YARN-9419
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9419.001.patch, YARN-9419.002.patch, 
> YARN-9419.003.patch, YARN-9419.branch-3.2.001.patch
>
>
> A WARN log should be added at least (logged once on startup) that notifies 
> the user about a potentially offending configuration: GPU isolation is 
> enabled but LCE is disabled.
> I think this is a dangerous, yet valid configuration: As LCE is the only 
> container executor that utilizes cgroups, no real HW-isolation happens if LCE 
> is disabled. 
> Let's suppose we have 2 GPU devices in 1 node:
>  # NM reports 2 devices (as a Resource) to RM
>  # RM assigns GPU#1 to container#2 that requests 1 GPU device
>  # When container#2 is also requesting 1 GPU device, RM is going to assign 
> either GPU#1 or GPU#2, so there's no guarantee that GPU#2 will be assigned. 
> If GPU#1 is assigned to a second container, nasty things could happen.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9419) Log a warning if GPU isolation is enabled but LinuxContainerExecutor is disabled

2020-03-09 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17055109#comment-17055109
 ] 

Andras Gyori commented on YARN-9419:


The cherry-pick was without errors, uploaded the backport patch.

> Log a warning if GPU isolation is enabled but LinuxContainerExecutor is 
> disabled
> 
>
> Key: YARN-9419
> URL: https://issues.apache.org/jira/browse/YARN-9419
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9419.001.patch, YARN-9419.002.patch, 
> YARN-9419.003.patch, YARN-9419.branch-3.2.001.patch
>
>
> A WARN log should be added at least (logged once on startup) that notifies 
> the user about a potentially offending configuration: GPU isolation is 
> enabled but LCE is disabled.
> I think this is a dangerous, yet valid configuration: As LCE is the only 
> container executor that utilizes cgroups, no real HW-isolation happens if LCE 
> is disabled. 
> Let's suppose we have 2 GPU devices in 1 node:
>  # NM reports 2 devices (as a Resource) to RM
>  # RM assigns GPU#1 to container#2 that requests 1 GPU device
>  # When container#2 is also requesting 1 GPU device, RM is going to assign 
> either GPU#1 or GPU#2, so there's no guarantee that GPU#2 will be assigned. 
> If GPU#1 is assigned to a second container, nasty things could happen.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-9997) Code cleanup in ZKConfigurationStore

2020-03-09 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054919#comment-17054919
 ] 

Andras Gyori edited comment on YARN-9997 at 3/9/20, 3:53 PM:
-

Thank you for your feedback, [~snemeth].

I have addressed all the issues, and reinforced them with additional unit 
tests. #logMutation should not overwrite the data when an invalid (non 
linked-list) object is queried from Zookeeper logpath. As for the third 
concern, receiving a null value as a config version is considered to be an 
invalid state and an IllegalStateException is thrown accordingly.


was (Author: gandras):
Thank you for feedback, [~snemeth].

I have addressed all the issues, and reinforced them with additional unit 
tests. #logMutation should not overwrite the data when an invalid (non 
linked-list) object is queried from Zookeeper logpath. As for the third 
concern, receiving a null value as a config version is considered to be an 
invalid state and an IllegalStateException is thrown accordingly.

> Code cleanup in ZKConfigurationStore
> 
>
> Key: YARN-9997
> URL: https://issues.apache.org/jira/browse/YARN-9997
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-9997.001.patch, YARN-9997.002.patch, 
> YARN-9997.003.patch, YARN-9997.004.patch
>
>
> Many thins can be improved:
> * znodeParentPath could be a local variable
> * zkManager could be private, VisibleForTesting annotation is not needed 
> anymore
> * Do something with unchecked casts
> * zkManager.safeSetData calls are almost having the same set of parameters: 
> Simplify this
> * Extract zkManager calls to their own methods: They are repeated
> * Remove TODOs



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10179) Queue mapping based on group id passed through application tag

2020-03-17 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10179:

Attachment: YARN-10179.001.patch

> Queue mapping based on group id passed through application tag
> --
>
> Key: YARN-10179
> URL: https://issues.apache.org/jira/browse/YARN-10179
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-10179.001.patch
>
>
> There are situations when the real submitting user differs from the user what 
> arrives to YARN. For example in case of a Hive application when Hive 
> impersonation is turned off, the hive queries will run as Hive user and the 
> mapping is done based on the user's group. 
> Unfortunately in this case YARN doesn't have any information about the real 
> user and there are cases when the customer may want to map these applications 
> to the real submitting user's queue (based on the group id) instead of the 
> Hive queue.
> For these cases, if they would pass the group id (or name) in the application 
> tag we may read it and use it during the queue mapping, if that user has 
> rights to run on the real user's queue.  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-10179) Queue mapping based on group id passed through application tag

2020-03-17 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori reassigned YARN-10179:
---

Assignee: Andras Gyori  (was: Szilard Nemeth)

> Queue mapping based on group id passed through application tag
> --
>
> Key: YARN-10179
> URL: https://issues.apache.org/jira/browse/YARN-10179
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Szilard Nemeth
>Assignee: Andras Gyori
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-10179.001.patch
>
>
> There are situations when the real submitting user differs from the user what 
> arrives to YARN. For example in case of a Hive application when Hive 
> impersonation is turned off, the hive queries will run as Hive user and the 
> mapping is done based on the user's group. 
> Unfortunately in this case YARN doesn't have any information about the real 
> user and there are cases when the customer may want to map these applications 
> to the real submitting user's queue (based on the group id) instead of the 
> Hive queue.
> For these cases, if they would pass the group id (or name) in the application 
> tag we may read it and use it during the queue mapping, if that user has 
> rights to run on the real user's queue.  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10179) Queue mapping based on group id passed through application tag

2020-03-17 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060730#comment-17060730
 ] 

Andras Gyori commented on YARN-10179:
-

Thank you, [~snemeth] for the references.
I have made an early version of the proposed changes, however there might be 
further questions about the requirements. As for now, the default behavior in 
case of an invalid tag/non-whitelisted tag is to fall back to the original 
source of mapping. Moreover, this approach only works if the user is a member 
of the group.

Might be interesting for [~lkovari], [~wilfreds]

> Queue mapping based on group id passed through application tag
> --
>
> Key: YARN-10179
> URL: https://issues.apache.org/jira/browse/YARN-10179
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-10179.001.patch
>
>
> There are situations when the real submitting user differs from the user what 
> arrives to YARN. For example in case of a Hive application when Hive 
> impersonation is turned off, the hive queries will run as Hive user and the 
> mapping is done based on the user's group. 
> Unfortunately in this case YARN doesn't have any information about the real 
> user and there are cases when the customer may want to map these applications 
> to the real submitting user's queue (based on the group id) instead of the 
> Hive queue.
> For these cases, if they would pass the group id (or name) in the application 
> tag we may read it and use it during the queue mapping, if that user has 
> rights to run on the real user's queue.  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10284) Add lazy initialization of LogAggregationFileControllerFactory in LogServlet

2020-05-27 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117660#comment-17117660
 ] 

Andras Gyori commented on YARN-10284:
-

Hey [~adam.antal]. Good job on finding this issue. I was wondering if the 
factory could be cached somehow? As far as I can see, it is reinitialized in 
every call. I understand, that it is error prone to initialize it in 
constructor, but it could be stored and reused if the initialization was 
successful.

> Add lazy initialization of LogAggregationFileControllerFactory in LogServlet
> 
>
> Key: YARN-10284
> URL: https://issues.apache.org/jira/browse/YARN-10284
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: log-aggregation, yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: YARN-10284.001.patch
>
>
> Suppose the {{mapred}} user has no access to the remote folder. Pinging the 
> JHS if it's online in every few seconds will produce the following entry in 
> the log:
> {noformat}
> 2020-05-19 00:17:20,331 WARN 
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController:
>  Unable to determine if the filesystem supports append operation
> java.nio.file.AccessDeniedException: test-bucket: 
> org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: There is no mapped role 
> for the group(s) associated with the authenticated user. (user: mapred)
>   at 
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:204)
> [...]
>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:513)
>   at 
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController.getRollOverLogMaxSize(LogAggregationIndexedFileController.java:1157)
>   at 
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController.initInternal(LogAggregationIndexedFileController.java:149)
>   at 
> org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController.initialize(LogAggregationFileController.java:135)
>   at 
> org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileControllerFactory.(LogAggregationFileControllerFactory.java:139)
>   at 
> org.apache.hadoop.yarn.server.webapp.LogServlet.(LogServlet.java:66)
>   at 
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.(HsWebServices.java:99)
>   at 
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices$$FastClassByGuice$$1eb8d5d6.newInstance()
>   at 
> com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
> [...]
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
>   at java.lang.Thread.run(Thread.java:748)
> {noformat}
> We should only create the {{LogAggregationFactory}} instance when we actually 
> need it, not every time the {{LogServlet}} object is instantiated (so 
> definitely not in the constructor). In this way we prevent pressure on the 
> S3A auth side, especially if the authentication request is a costly operation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-10304) Create an endpoint for remote application log directory path query

2020-06-02 Thread Andras Gyori (Jira)
Andras Gyori created YARN-10304:
---

 Summary: Create an endpoint for remote application log directory 
path query
 Key: YARN-10304
 URL: https://issues.apache.org/jira/browse/YARN-10304
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Andras Gyori
Assignee: Andras Gyori


The logic of the aggregated log directory path determination (currently based 
on configuration) is scattered around the codebase and duplicated multiple 
times. By providing a separate class for creating the path for a specific user, 
it allows for an abstraction over this logic. This could be used in place of 
the previously duplicated logic, moreover, we could provide an endpoint to 
query this path.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10304) Create an endpoint for remote application log directory path query

2020-06-02 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123405#comment-17123405
 ] 

Andras Gyori commented on YARN-10304:
-

An early draft patch has been uploaded, that provides the foundation of the 
logic. As for now, it only contains the bare minimum of the path creation, 
without involving any bucket or group based naming convention. Whether to 
include anything more is up for discussion.

> Create an endpoint for remote application log directory path query
> --
>
> Key: YARN-10304
> URL: https://issues.apache.org/jira/browse/YARN-10304
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10304.001.patch
>
>
> The logic of the aggregated log directory path determination (currently based 
> on configuration) is scattered around the codebase and duplicated multiple 
> times. By providing a separate class for creating the path for a specific 
> user, it allows for an abstraction over this logic. This could be used in 
> place of the previously duplicated logic, moreover, we could provide an 
> endpoint to query this path.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10304) Create an endpoint for remote application log directory path query

2020-06-02 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10304:

Attachment: YARN-10304.001.patch

> Create an endpoint for remote application log directory path query
> --
>
> Key: YARN-10304
> URL: https://issues.apache.org/jira/browse/YARN-10304
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10304.001.patch
>
>
> The logic of the aggregated log directory path determination (currently based 
> on configuration) is scattered around the codebase and duplicated multiple 
> times. By providing a separate class for creating the path for a specific 
> user, it allows for an abstraction over this logic. This could be used in 
> place of the previously duplicated logic, moreover, we could provide an 
> endpoint to query this path.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10284) Add lazy initialization of LogAggregationFileControllerFactory in LogServlet

2020-06-02 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123426#comment-17123426
 ] 

Andras Gyori commented on YARN-10284:
-

Last patch LGTM (non-binding), I like the caching approach +1.

> Add lazy initialization of LogAggregationFileControllerFactory in LogServlet
> 
>
> Key: YARN-10284
> URL: https://issues.apache.org/jira/browse/YARN-10284
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: log-aggregation, yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: YARN-10284.001.patch, YARN-10284.002.patch, 
> YARN-10284.003.patch, YARN-10284.004.patch
>
>
> Suppose the {{mapred}} user has no access to the remote folder. Pinging the 
> JHS if it's online in every few seconds will produce the following entry in 
> the log:
> {noformat}
> 2020-05-19 00:17:20,331 WARN 
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController:
>  Unable to determine if the filesystem supports append operation
> java.nio.file.AccessDeniedException: test-bucket: 
> org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: There is no mapped role 
> for the group(s) associated with the authenticated user. (user: mapred)
>   at 
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:204)
> [...]
>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:513)
>   at 
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController.getRollOverLogMaxSize(LogAggregationIndexedFileController.java:1157)
>   at 
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController.initInternal(LogAggregationIndexedFileController.java:149)
>   at 
> org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController.initialize(LogAggregationFileController.java:135)
>   at 
> org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileControllerFactory.(LogAggregationFileControllerFactory.java:139)
>   at 
> org.apache.hadoop.yarn.server.webapp.LogServlet.(LogServlet.java:66)
>   at 
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.(HsWebServices.java:99)
>   at 
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices$$FastClassByGuice$$1eb8d5d6.newInstance()
>   at 
> com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
> [...]
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
>   at java.lang.Thread.run(Thread.java:748)
> {noformat}
> We should only create the {{LogAggregationFactory}} instance when we actually 
> need it, not every time the {{LogServlet}} object is instantiated (so 
> definitely not in the constructor). In this way we prevent pressure on the 
> S3A auth side, especially if the authentication request is a costly operation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10286) PendingContainers bugs in the scheduler outputs

2020-06-02 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10286:

Attachment: YARN-10286.branch-3.3.001.patch

> PendingContainers bugs in the scheduler outputs
> ---
>
> Key: YARN-10286
> URL: https://issues.apache.org/jira/browse/YARN-10286
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Critical
> Fix For: 3.4.0
>
> Attachments: YARN-10286.001.patch, YARN-10286.branch-3.2.001.patch, 
> YARN-10286.branch-3.3.001.patch
>
>
> There are some problems with the  {{ws/v1/cluster/scheduler}} output of the 
> ResourceManager:
> Even though the pendingContainers field is 
> [documented|https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API]
>  in the FairScheduler output, it is 
> [missing|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerQueueInfo.java]
>  from the actual output. In case of the CapacityScheduler this field 
> [exists|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java],
>  but it is missing from the documentation. Let's fix both cases!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10286) PendingContainers bugs in the scheduler outputs

2020-06-02 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10286:

Attachment: YARN-10286.branch-3.2.001.patch

> PendingContainers bugs in the scheduler outputs
> ---
>
> Key: YARN-10286
> URL: https://issues.apache.org/jira/browse/YARN-10286
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Critical
> Fix For: 3.4.0
>
> Attachments: YARN-10286.001.patch, YARN-10286.branch-3.2.001.patch, 
> YARN-10286.branch-3.3.001.patch
>
>
> There are some problems with the  {{ws/v1/cluster/scheduler}} output of the 
> ResourceManager:
> Even though the pendingContainers field is 
> [documented|https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API]
>  in the FairScheduler output, it is 
> [missing|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerQueueInfo.java]
>  from the actual output. In case of the CapacityScheduler this field 
> [exists|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java],
>  but it is missing from the documentation. Let's fix both cases!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10286) PendingContainers bugs in the scheduler outputs

2020-06-02 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123910#comment-17123910
 ] 

Andras Gyori commented on YARN-10286:
-

The patch smoothly applied to both branch-3.3 and 3.2.

> PendingContainers bugs in the scheduler outputs
> ---
>
> Key: YARN-10286
> URL: https://issues.apache.org/jira/browse/YARN-10286
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Critical
> Fix For: 3.4.0
>
> Attachments: YARN-10286.001.patch, YARN-10286.branch-3.2.001.patch, 
> YARN-10286.branch-3.3.001.patch
>
>
> There are some problems with the  {{ws/v1/cluster/scheduler}} output of the 
> ResourceManager:
> Even though the pendingContainers field is 
> [documented|https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API]
>  in the FairScheduler output, it is 
> [missing|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerQueueInfo.java]
>  from the actual output. In case of the CapacityScheduler this field 
> [exists|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java],
>  but it is missing from the documentation. Let's fix both cases!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-4783) Log aggregation failure for application when Nodemanager is restarted

2020-07-30 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-4783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori reassigned YARN-4783:
--

Assignee: Andras Gyori

> Log aggregation failure for application when Nodemanager is restarted 
> --
>
> Key: YARN-4783
> URL: https://issues.apache.org/jira/browse/YARN-4783
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Andras Gyori
>Priority: Major
>
> Scenario :
> =
> 1.Start NM with user dsperf:hadoop
> 2.Configure linux-execute user as dsperf
> 3.Submit application with yarn user 
> 4.Once few containers are allocated to NM 1
> 5.Nodemanager 1 is stopped  (wait for expiry )
> 6.Start node manager after application is completed
> 7.Check the log aggregation is happening for the containers log in NMLocal 
> directory
> Expect Output :
> ===
> Log aggregation should be succesfull
> Actual Output :
> ===
> Log aggreation not successfull



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-4783) Log aggregation failure for application when Nodemanager is restarted

2020-07-30 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-4783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-4783:
---
Attachment: YARN-4783.001.patch

> Log aggregation failure for application when Nodemanager is restarted 
> --
>
> Key: YARN-4783
> URL: https://issues.apache.org/jira/browse/YARN-4783
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-4783.001.patch
>
>
> Scenario :
> =
> 1.Start NM with user dsperf:hadoop
> 2.Configure linux-execute user as dsperf
> 3.Submit application with yarn user 
> 4.Once few containers are allocated to NM 1
> 5.Nodemanager 1 is stopped  (wait for expiry )
> 6.Start node manager after application is completed
> 7.Check the log aggregation is happening for the containers log in NMLocal 
> directory
> Expect Output :
> ===
> Log aggregation should be succesfull
> Actual Output :
> ===
> Log aggreation not successfull



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Reopened] (YARN-4783) Log aggregation failure for application when Nodemanager is restarted

2020-07-30 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-4783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori reopened YARN-4783:


I am reopening this issue in order to find a less invasive approach on how to 
handle this corner case, since it was reported a long time ago and still has 
not been resolved yet.
Uploaded a new patch without a test case for now.

The main idea is to try to renew the token stored in the application 
credentials, on an application state transition from NEW to INITING. If the 
renewal process is successful, the token is valid and nothing needs to be done 
from the application's point of view. However, if the renewal is failed with 
InvalidToken error, we request a new one on behalf of the user.

In case of a token request, it is now the application's responsibility to clean 
it up, when the corresponding operations are done, therefore it is canceled 
when the log aggregation is finished.

> Log aggregation failure for application when Nodemanager is restarted 
> --
>
> Key: YARN-4783
> URL: https://issues.apache.org/jira/browse/YARN-4783
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Andras Gyori
>Priority: Major
>
> Scenario :
> =
> 1.Start NM with user dsperf:hadoop
> 2.Configure linux-execute user as dsperf
> 3.Submit application with yarn user 
> 4.Once few containers are allocated to NM 1
> 5.Nodemanager 1 is stopped  (wait for expiry )
> 6.Start node manager after application is completed
> 7.Check the log aggregation is happening for the containers log in NMLocal 
> directory
> Expect Output :
> ===
> Log aggregation should be succesfull
> Actual Output :
> ===
> Log aggreation not successfull



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10304) Create an endpoint for remote application log directory path query

2020-07-31 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10304:

Attachment: YARN-10304.008.patch

> Create an endpoint for remote application log directory path query
> --
>
> Key: YARN-10304
> URL: https://issues.apache.org/jira/browse/YARN-10304
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10304.001.patch, YARN-10304.002.patch, 
> YARN-10304.003.patch, YARN-10304.004.patch, YARN-10304.005.patch, 
> YARN-10304.006.patch, YARN-10304.007.patch, YARN-10304.008.patch
>
>
> The logic of the aggregated log directory path determination (currently based 
> on configuration) is scattered around the codebase and duplicated multiple 
> times. By providing a separate class for creating the path for a specific 
> user, it allows for an abstraction over this logic. This could be used in 
> place of the previously duplicated logic, moreover, we could provide an 
> endpoint to query this path.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10342) [UI1] Provide a way to hide Tools section in Web UIv1

2020-08-06 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10342:

Attachment: YARN-10342.003.patch

> [UI1] Provide a way to hide Tools section in Web UIv1
> -
>
> Key: YARN-10342
> URL: https://issues.apache.org/jira/browse/YARN-10342
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: Screenshot 2020-07-09 at 14.13.19.png, 
> YARN-10342.001.patch, YARN-10342.002.patch, YARN-10342.003.patch
>
>
> The Tools section in web UI1 might contain sensitive information, which 
> should ideally be hidden from end users. We should provide a configurable 
> value to hide it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10342) [UI1] Provide a way to hide Tools section in Web UIv1

2020-08-06 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17172144#comment-17172144
 ] 

Andras Gyori commented on YARN-10342:
-

Thank you [~BilwaST] for the feedback, a good catch, fixed the appropriate 
classes to inject Configuration object in the constructor.

> [UI1] Provide a way to hide Tools section in Web UIv1
> -
>
> Key: YARN-10342
> URL: https://issues.apache.org/jira/browse/YARN-10342
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: Screenshot 2020-07-09 at 14.13.19.png, 
> YARN-10342.001.patch, YARN-10342.002.patch, YARN-10342.003.patch
>
>
> The Tools section in web UI1 might contain sensitive information, which 
> should ideally be hidden from end users. We should provide a configurable 
> value to hide it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-4783) Log aggregation failure for application when Nodemanager is restarted

2020-08-06 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-4783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-4783:
---
Attachment: YARN-4783.003.patch

> Log aggregation failure for application when Nodemanager is restarted 
> --
>
> Key: YARN-4783
> URL: https://issues.apache.org/jira/browse/YARN-4783
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-4783.001.patch, YARN-4783.002.patch, 
> YARN-4783.003.patch
>
>
> Scenario :
> =
> 1.Start NM with user dsperf:hadoop
> 2.Configure linux-execute user as dsperf
> 3.Submit application with yarn user 
> 4.Once few containers are allocated to NM 1
> 5.Nodemanager 1 is stopped  (wait for expiry )
> 6.Start node manager after application is completed
> 7.Check the log aggregation is happening for the containers log in NMLocal 
> directory
> Expect Output :
> ===
> Log aggregation should be succesfull
> Actual Output :
> ===
> Log aggreation not successfull



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10318) ApplicationHistory Web UI incorrect column indexing

2020-06-30 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10318:

Fix Version/s: 3.3.0

> ApplicationHistory Web UI incorrect column indexing
> ---
>
> Key: YARN-10318
> URL: https://issues.apache.org/jira/browse/YARN-10318
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Fix For: 3.3.0, 3.4.0
>
> Attachments: Screenshot 2020-06-25 at 10.15.32.png, 
> YARN-10318.001.patch, image-2020-06-16-17-14-55-921.png
>
>
> The ApplicationHistory UI is broken due to an incorrect column indexing. This 
> bug was probably introduced in YARN-10038, which presumes, that the table 
> contains the application tag column (which is true for RM Web UI, but not for 
> AH Web UI).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10318) ApplicationHistory Web UI incorrect column indexing

2020-06-30 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148400#comment-17148400
 ] 

Andras Gyori commented on YARN-10318:
-

Thank you [~snemeth] for the commit. I have checked, and added 3.3 as a 
backport target as well.

> ApplicationHistory Web UI incorrect column indexing
> ---
>
> Key: YARN-10318
> URL: https://issues.apache.org/jira/browse/YARN-10318
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: Screenshot 2020-06-25 at 10.15.32.png, 
> YARN-10318.001.patch, image-2020-06-16-17-14-55-921.png
>
>
> The ApplicationHistory UI is broken due to an incorrect column indexing. This 
> bug was probably introduced in YARN-10038, which presumes, that the table 
> contains the application tag column (which is true for RM Web UI, but not for 
> AH Web UI).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10318) ApplicationHistory Web UI incorrect column indexing

2020-06-30 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10318:

Attachment: YARN-10318.branch-3.3.001.patch

> ApplicationHistory Web UI incorrect column indexing
> ---
>
> Key: YARN-10318
> URL: https://issues.apache.org/jira/browse/YARN-10318
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Fix For: 3.3.0, 3.4.0
>
> Attachments: Screenshot 2020-06-25 at 10.15.32.png, 
> YARN-10318.001.patch, YARN-10318.branch-3.3.001.patch, 
> image-2020-06-16-17-14-55-921.png
>
>
> The ApplicationHistory UI is broken due to an incorrect column indexing. This 
> bug was probably introduced in YARN-10038, which presumes, that the table 
> contains the application tag column (which is true for RM Web UI, but not for 
> AH Web UI).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10304) Create an endpoint for remote application log directory path query

2020-07-01 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149485#comment-17149485
 ] 

Andras Gyori commented on YARN-10304:
-

Thank you [~mhudaky] for the review, good catch! I have uploaded a new patch 
that incorporates this idea.

> Create an endpoint for remote application log directory path query
> --
>
> Key: YARN-10304
> URL: https://issues.apache.org/jira/browse/YARN-10304
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10304.001.patch, YARN-10304.002.patch, 
> YARN-10304.003.patch, YARN-10304.004.patch, YARN-10304.005.patch, 
> YARN-10304.006.patch, YARN-10304.007.patch
>
>
> The logic of the aggregated log directory path determination (currently based 
> on configuration) is scattered around the codebase and duplicated multiple 
> times. By providing a separate class for creating the path for a specific 
> user, it allows for an abstraction over this logic. This could be used in 
> place of the previously duplicated logic, moreover, we could provide an 
> endpoint to query this path.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10304) Create an endpoint for remote application log directory path query

2020-07-01 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10304:

Attachment: YARN-10304.007.patch

> Create an endpoint for remote application log directory path query
> --
>
> Key: YARN-10304
> URL: https://issues.apache.org/jira/browse/YARN-10304
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10304.001.patch, YARN-10304.002.patch, 
> YARN-10304.003.patch, YARN-10304.004.patch, YARN-10304.005.patch, 
> YARN-10304.006.patch, YARN-10304.007.patch
>
>
> The logic of the aggregated log directory path determination (currently based 
> on configuration) is scattered around the codebase and duplicated multiple 
> times. By providing a separate class for creating the path for a specific 
> user, it allows for an abstraction over this logic. This could be used in 
> place of the previously duplicated logic, moreover, we could provide an 
> endpoint to query this path.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10342) [UI1] Provide a way to hide Tools section in Web UIv1

2020-07-09 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154502#comment-17154502
 ] 

Andras Gyori commented on YARN-10342:
-

A patch has been uploaded, which puts a constraint on the tools section in 
applicationhistory, nodemanager, resourcemanager and mapreduce v2 historyserver 
webapp. It is vital to emphasize, however, that the links are still accessible 
in case the user knows the address (like /conf or /logs).

> [UI1] Provide a way to hide Tools section in Web UIv1
> -
>
> Key: YARN-10342
> URL: https://issues.apache.org/jira/browse/YARN-10342
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10342.001.patch
>
>
> The Tools section in web UI1 might contain sensitive information, which 
> should ideally be hidden from end users. We should provide a configurable 
> value to hide it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10342) [UI1] Provide a way to hide Tools section in Web UIv1

2020-07-09 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154502#comment-17154502
 ] 

Andras Gyori edited comment on YARN-10342 at 7/9/20, 12:36 PM:
---

A patch has been uploaded, which puts a constraint on the tools section in 
applicationhistory, nodemanager, resourcemanager and mapreduce v2 historyserver 
webapp. It is vital to emphasize, however, that the links are still accessible 
in case the user knows the address (like /conf or /logs). The name of the 
configuration value is:
{noformat}
yarn.webapp.ui1.tools.enable{noformat}


was (Author: gandras):
A patch has been uploaded, which puts a constraint on the tools section in 
applicationhistory, nodemanager, resourcemanager and mapreduce v2 historyserver 
webapp. It is vital to emphasize, however, that the links are still accessible 
in case the user knows the address (like /conf or /logs).

> [UI1] Provide a way to hide Tools section in Web UIv1
> -
>
> Key: YARN-10342
> URL: https://issues.apache.org/jira/browse/YARN-10342
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10342.001.patch
>
>
> The Tools section in web UI1 might contain sensitive information, which 
> should ideally be hidden from end users. We should provide a configurable 
> value to hide it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10342) [UI1] Provide a way to hide Tools section in Web UIv1

2020-07-09 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10342:

Attachment: YARN-10342.001.patch

> [UI1] Provide a way to hide Tools section in Web UIv1
> -
>
> Key: YARN-10342
> URL: https://issues.apache.org/jira/browse/YARN-10342
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10342.001.patch
>
>
> The Tools section in web UI1 might contain sensitive information, which 
> should ideally be hidden from end users. We should provide a configurable 
> value to hide it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10342) [UI1] Provide a way to hide Tools section in Web UIv1

2020-07-09 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10342:

Attachment: Screenshot 2020-07-09 at 14.13.19.png

> [UI1] Provide a way to hide Tools section in Web UIv1
> -
>
> Key: YARN-10342
> URL: https://issues.apache.org/jira/browse/YARN-10342
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: Screenshot 2020-07-09 at 14.13.19.png, 
> YARN-10342.001.patch
>
>
> The Tools section in web UI1 might contain sensitive information, which 
> should ideally be hidden from end users. We should provide a configurable 
> value to hide it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8631) YARN RM fails to add the application to the delegation token renewer on recovery

2020-07-09 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154737#comment-17154737
 ] 

Andras Gyori commented on YARN-8631:


[~umittal] I was trying to reproduce the issue, however, the attached patch is 
already outdated. Could you give a more up-to-date example please?

> YARN RM fails to add the application to the delegation token renewer on 
> recovery
> 
>
> Key: YARN-8631
> URL: https://issues.apache.org/jira/browse/YARN-8631
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.1.0
>Reporter: Sanjay Divgi
>Assignee: Umesh Mittal
>Priority: Blocker
> Attachments: YARN-8631.001.patch, 
> hadoop-yarn-resourcemanager-ctr-e138-1518143905142-429059-01-04.log
>
>
> On HA cluster we have observed that yarn resource manager fails to add the 
> application to the delegation token renewer on recovery.
> Below is the error:
> {code:java}
> 2018-08-07 08:41:23,850 INFO security.DelegationTokenRenewer 
> (DelegationTokenRenewer.java:renewToken(635)) - Renewed delegation-token= 
> [Kind: TIMELINE_DELEGATION_TOKEN, Service: 172.27.84.192:8188, Ident: 
> (TIMELINE_DELEGATION_TOKEN owner=hrt_qa_hive_spark, renewer=yarn, realUser=, 
> issueDate=1533624642302, maxDate=1534229442302, sequenceNumber=18, 
> masterKeyId=4);exp=1533717683478; apps=[application_1533623972681_0001]]
> 2018-08-07 08:41:23,855 WARN security.DelegationTokenRenewer 
> (DelegationTokenRenewer.java:handleDTRenewerAppRecoverEvent(955)) - Unable to 
> add the application to the delegation token renewer on recovery.
> java.lang.NullPointerException
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:522)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleDTRenewerAppRecoverEvent(DelegationTokenRenewer.java:953)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$700(DelegationTokenRenewer.java:79)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:912)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-10342) [UI1] Provide a way to hide Tools section in Web UIv1

2020-07-07 Thread Andras Gyori (Jira)
Andras Gyori created YARN-10342:
---

 Summary: [UI1] Provide a way to hide Tools section in Web UIv1
 Key: YARN-10342
 URL: https://issues.apache.org/jira/browse/YARN-10342
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Andras Gyori
Assignee: Andras Gyori


The Tools section in web UI1 might contain sensitive information, which should 
ideally be hidden from end users. We should provide a configurable value to 
hide it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10342) [UI1] Provide a way to hide Tools section in Web UIv1

2020-07-15 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158094#comment-17158094
 ] 

Andras Gyori commented on YARN-10342:
-

Thank you [~bteke] for the review! Though I would agree with your point, I 
think HtmlBlock is a mix-in class, that is used in different scenarios than 
NavBlock as well. I am afraid, that _showTools_ would not be useful for those 
classes, and probably would cause confusion.

> [UI1] Provide a way to hide Tools section in Web UIv1
> -
>
> Key: YARN-10342
> URL: https://issues.apache.org/jira/browse/YARN-10342
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: Screenshot 2020-07-09 at 14.13.19.png, 
> YARN-10342.001.patch
>
>
> The Tools section in web UI1 might contain sensitive information, which 
> should ideally be hidden from end users. We should provide a configurable 
> value to hide it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10325) Document max-parallel-apps for Capacity Scheduler

2020-06-23 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142649#comment-17142649
 ] 

Andras Gyori commented on YARN-10325:
-

The patch LGTM, +1.

> Document max-parallel-apps for Capacity Scheduler
> -
>
> Key: YARN-10325
> URL: https://issues.apache.org/jira/browse/YARN-10325
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, capacityscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10325-001.patch
>
>
> New feature introduced by YARN-9930 should be reflected in the upstream 
> documentation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10304) Create an endpoint for remote application log directory path query

2020-06-23 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142847#comment-17142847
 ] 

Andras Gyori commented on YARN-10304:
-

Thank you [~adam.antal] for the review. I have addressed all the remaining 
concerns. Extended the logic with the application specific path assembly, and 
added 2 more unit tests to cover the scenarios.

> Create an endpoint for remote application log directory path query
> --
>
> Key: YARN-10304
> URL: https://issues.apache.org/jira/browse/YARN-10304
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10304.001.patch, YARN-10304.002.patch, 
> YARN-10304.003.patch, YARN-10304.004.patch, YARN-10304.005.patch, 
> YARN-10304.006.patch
>
>
> The logic of the aggregated log directory path determination (currently based 
> on configuration) is scattered around the codebase and duplicated multiple 
> times. By providing a separate class for creating the path for a specific 
> user, it allows for an abstraction over this logic. This could be used in 
> place of the previously duplicated logic, moreover, we could provide an 
> endpoint to query this path.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10304) Create an endpoint for remote application log directory path query

2020-06-23 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10304:

Attachment: YARN-10304.006.patch

> Create an endpoint for remote application log directory path query
> --
>
> Key: YARN-10304
> URL: https://issues.apache.org/jira/browse/YARN-10304
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10304.001.patch, YARN-10304.002.patch, 
> YARN-10304.003.patch, YARN-10304.004.patch, YARN-10304.005.patch, 
> YARN-10304.006.patch
>
>
> The logic of the aggregated log directory path determination (currently based 
> on configuration) is scattered around the codebase and duplicated multiple 
> times. By providing a separate class for creating the path for a specific 
> user, it allows for an abstraction over this logic. This could be used in 
> place of the previously duplicated logic, moreover, we could provide an 
> endpoint to query this path.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10327) Remove duplication of checking for invalid application ID in TestLogsCLI

2020-06-25 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145010#comment-17145010
 ] 

Andras Gyori commented on YARN-10327:
-

Thank you for working on this issue [~mhudaky]. The patch looks good to me. 
Following the same logic, there are invalidOpts as well, that could belong to 
testLogsCLIWithInvalidArgs method. However, for this issue, I think it is 
enough.

> Remove duplication of checking for invalid application ID in TestLogsCLI
> 
>
> Key: YARN-10327
> URL: https://issues.apache.org/jira/browse/YARN-10327
> Project: Hadoop YARN
>  Issue Type: Test
>Reporter: Hudáky Márton Gyula
>Assignee: Hudáky Márton Gyula
>Priority: Trivial
> Attachments: YARN-10327.001.patch
>
>
> TestLogsCLI has a separate function to test for invalid application ID 
> (#testInvalidApplicationId) and another (#testLogsCLIWithInvalidArgs) to test 
> multiple invalid arguments (including application ID). One of them should be 
> eliminated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10327) Remove duplication of checking for invalid application ID in TestLogsCLI

2020-06-26 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145010#comment-17145010
 ] 

Andras Gyori edited comment on YARN-10327 at 6/26/20, 9:26 AM:
---

Thank you for working on this issue [~mhudaky]. The patch looks good to me, +1 
non binding. Following the same logic, there are invalidOpts as well, that 
could belong to testLogsCLIWithInvalidArgs method. However, for this issue, I 
think it is enough.


was (Author: gandras):
Thank you for working on this issue [~mhudaky]. The patch looks good to me. 
Following the same logic, there are invalidOpts as well, that could belong to 
testLogsCLIWithInvalidArgs method. However, for this issue, I think it is 
enough.

> Remove duplication of checking for invalid application ID in TestLogsCLI
> 
>
> Key: YARN-10327
> URL: https://issues.apache.org/jira/browse/YARN-10327
> Project: Hadoop YARN
>  Issue Type: Test
>Reporter: Hudáky Márton Gyula
>Assignee: Hudáky Márton Gyula
>Priority: Trivial
> Attachments: YARN-10327.001.patch
>
>
> TestLogsCLI has a separate function to test for invalid application ID 
> (#testInvalidApplicationId) and another (#testLogsCLIWithInvalidArgs) to test 
> multiple invalid arguments (including application ID). One of them should be 
> eliminated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-10318) ApplicationHistory Web UI incorrect column indexing

2020-06-16 Thread Andras Gyori (Jira)
Andras Gyori created YARN-10318:
---

 Summary: ApplicationHistory Web UI incorrect column indexing
 Key: YARN-10318
 URL: https://issues.apache.org/jira/browse/YARN-10318
 Project: Hadoop YARN
  Issue Type: Bug
  Components: yarn
Reporter: Andras Gyori
Assignee: Andras Gyori
 Attachments: image-2020-06-16-17-14-55-921.png

The ApplicationHistory UI is broken due to an incorrect column indexing. This 
bug was probably introduced in YARN-10038, which presumes, that the table 
contains the application tag column (which is true for RM Web UI, but not for 
AH Web UI).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10304) Create an endpoint for remote application log directory path query

2020-06-15 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10304:

Attachment: YARN-10304.005.patch

> Create an endpoint for remote application log directory path query
> --
>
> Key: YARN-10304
> URL: https://issues.apache.org/jira/browse/YARN-10304
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10304.001.patch, YARN-10304.002.patch, 
> YARN-10304.003.patch, YARN-10304.004.patch, YARN-10304.005.patch
>
>
> The logic of the aggregated log directory path determination (currently based 
> on configuration) is scattered around the codebase and duplicated multiple 
> times. By providing a separate class for creating the path for a specific 
> user, it allows for an abstraction over this logic. This could be used in 
> place of the previously duplicated logic, moreover, we could provide an 
> endpoint to query this path.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10304) Create an endpoint for remote application log directory path query

2020-06-15 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135700#comment-17135700
 ] 

Andras Gyori commented on YARN-10304:
-

Fixed the remaining checkstyle issues with the latest patch.

> Create an endpoint for remote application log directory path query
> --
>
> Key: YARN-10304
> URL: https://issues.apache.org/jira/browse/YARN-10304
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10304.001.patch, YARN-10304.002.patch, 
> YARN-10304.003.patch, YARN-10304.004.patch, YARN-10304.005.patch
>
>
> The logic of the aggregated log directory path determination (currently based 
> on configuration) is scattered around the codebase and duplicated multiple 
> times. By providing a separate class for creating the path for a specific 
> user, it allows for an abstraction over this logic. This could be used in 
> place of the previously duplicated logic, moreover, we could provide an 
> endpoint to query this path.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-10309) Dump scheduler and queue state information into CapacityScheduler statedump

2020-06-15 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori reassigned YARN-10309:
---

Assignee: Andras Gyori  (was: Prabhu Joseph)

> Dump scheduler and queue state information into CapacityScheduler statedump 
> 
>
> Key: YARN-10309
> URL: https://issues.apache.org/jira/browse/YARN-10309
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacityscheduler
>Reporter: Prabhu Joseph
>Assignee: Andras Gyori
>Priority: Major
>
> CapacityScheduler dump with Scheduler, FiCaSchedulerNode, FiCaSchedulerApp, 
> ParentQueue, LeafQueue, app and queue level ordering, multi node lookup 
> ordering details will make it ease to debug scheduler related issues instead 
> of correlating Debug Logs with CapacityScheduler code to get values for the 
> above.
> This is similar to FairScheduler statedump YARN-6042.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10309) Dump scheduler and queue state information into CapacityScheduler statedump

2020-06-15 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135896#comment-17135896
 ] 

Andras Gyori commented on YARN-10309:
-

Uploaded a draft patch of this feature. Waiting for further input to extend it 
with additional informations.

> Dump scheduler and queue state information into CapacityScheduler statedump 
> 
>
> Key: YARN-10309
> URL: https://issues.apache.org/jira/browse/YARN-10309
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacityscheduler
>Reporter: Prabhu Joseph
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10309.001.patch
>
>
> CapacityScheduler dump with Scheduler, FiCaSchedulerNode, FiCaSchedulerApp, 
> ParentQueue, LeafQueue, app and queue level ordering, multi node lookup 
> ordering details will make it ease to debug scheduler related issues instead 
> of correlating Debug Logs with CapacityScheduler code to get values for the 
> above.
> This is similar to FairScheduler statedump YARN-6042.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10309) Dump scheduler and queue state information into CapacityScheduler statedump

2020-06-15 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10309:

Attachment: YARN-10309.001.patch

> Dump scheduler and queue state information into CapacityScheduler statedump 
> 
>
> Key: YARN-10309
> URL: https://issues.apache.org/jira/browse/YARN-10309
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacityscheduler
>Reporter: Prabhu Joseph
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10309.001.patch
>
>
> CapacityScheduler dump with Scheduler, FiCaSchedulerNode, FiCaSchedulerApp, 
> ParentQueue, LeafQueue, app and queue level ordering, multi node lookup 
> ordering details will make it ease to debug scheduler related issues instead 
> of correlating Debug Logs with CapacityScheduler code to get values for the 
> above.
> This is similar to FairScheduler statedump YARN-6042.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10316) FS-CS converter: convert maxAppsDefault, maxRunningApps settings

2020-06-22 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142124#comment-17142124
 ] 

Andras Gyori commented on YARN-10316:
-

Last patch LGTM +1. Out of curiosity, I was wondering what happens, when 
max-parallel-apps is defined as a negative number. I have checked, that the 
converter correctly maps this to the same negative number. Does FS handle it 
the same as CS does?

> FS-CS converter: convert maxAppsDefault, maxRunningApps settings
> 
>
> Key: YARN-10316
> URL: https://issues.apache.org/jira/browse/YARN-10316
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>  Labels: fs2cs
> Attachments: YARN-10136-001.patch, YARN-10316-002.patch, 
> YARN-10316-003.patch
>
>
> In YARN-9930, support for maximum running applications (called "max parallel 
> apps") has been introduced.
> The converter now can handle the following settings in {{fair-scheduler.xml}}:
>  * {{}} per user
>  * {{}} per queue
>  * {{}} 
>  * {{}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10318) ApplicationHistory Web UI incorrect column indexing

2020-06-25 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10318:

Attachment: Screenshot 2020-06-25 at 10.15.32.png

> ApplicationHistory Web UI incorrect column indexing
> ---
>
> Key: YARN-10318
> URL: https://issues.apache.org/jira/browse/YARN-10318
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: Screenshot 2020-06-25 at 10.15.32.png, 
> image-2020-06-16-17-14-55-921.png
>
>
> The ApplicationHistory UI is broken due to an incorrect column indexing. This 
> bug was probably introduced in YARN-10038, which presumes, that the table 
> contains the application tag column (which is true for RM Web UI, but not for 
> AH Web UI).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



  1   2   >