[jira] [Assigned] (YARN-9097) Investigate why GpuDiscoverer methods are synchronized
[ https://issues.apache.org/jira/browse/YARN-9097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton reassigned YARN-9097: -- Assignee: (was: Kinga Marton) > Investigate why GpuDiscoverer methods are synchronized > -- > > Key: YARN-9097 > URL: https://issues.apache.org/jira/browse/YARN-9097 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Priority: Minor > Labels: newbie, newbie++ > > GpuDiscoverer.initialize surely shouldn't have been synchronized. > Please also investigate why getGpuDeviceInformation / getGpusUsableByYarn are > synchronized. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9371) Better logging for initialization of cgroups in CGroupsHandlerImpl
[ https://issues.apache.org/jira/browse/YARN-9371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton reassigned YARN-9371: -- Assignee: (was: Kinga Marton) > Better logging for initialization of cgroups in CGroupsHandlerImpl > -- > > Key: YARN-9371 > URL: https://issues.apache.org/jira/browse/YARN-9371 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Priority: Major > > We already have YARN-9098 that dealt with (not yet merged, though) code > duplication in cgroups setup code. > However, there's still room for improvement in the area of cgroups > initialization, especially in CGroupsHandlerImpl. > 1. > org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsHandlerImpl#initializeControllerPaths: > There's a decision in this method: Either cgroups are parsed from the > configured mount path or from the mounted cgroups descriptor file > (/proc/mounts). This decisions needs to be logged, along with some details, > e.g. cgroups mount path. > 2. In ResourceHandlerModule#parseConfiguredCGroupPath: Every candidate file > could be logged to debug. Moreover, the returned pathSubsystemMappings could > be also logged to info. > 3. In CGroupsHandlerImpl#parseMtab: This method iterates over lines of the > mtab file, tries to match the line and parses data of cgroups. > We need to better have logs for: > - If the line does not match: Print the line (I think debug level is fine for > that) > - We could log the entire result map to info log. > 4. In CGroupsHandlerImpl#initializeControllerPaths, > initializeControllerPathsFromMtab is called. This method returns mapping of > cgroups controllers (cpu, memory, devices, etc.) to file system paths. This > is an important piece of information for troubleshooting, so the map needs to > be logged to info. > 5. In CGroupsHandlerImpl#initializeCGroupController: There's a condition for > enableCGroupMount, I think it's worth to log if we are dealing with > YARN-driven cgroups mount or pre-mounted controllers. > 6. In CGroupsHandlerImpl#mountCGroupController: > - existingMountPath needs to be logged for better troubleshooting > - Other debug logs could also be added, this is up to the owner of this jira. > 7. In CGroupsHandlerImpl#initializePreMountedCGroupController: > - controllerPath needs to be logged for better troubleshooting. > - Other debug logs could also be added, this is up to the owner of this jira. > > A side-note: Implementing this Jira could be way easier after YARN-9098 is > merged as that Jira eliminated lots of code duplication and made the > initialization of cgroups more clear and concise. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7145) Identify potential flaky unit tests
[ https://issues.apache.org/jira/browse/YARN-7145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton reassigned YARN-7145: -- Assignee: (was: Kinga Marton) > Identify potential flaky unit tests > --- > > Key: YARN-7145 > URL: https://issues.apache.org/jira/browse/YARN-7145 > Project: Hadoop YARN > Issue Type: Test > Components: nodemanager, resourcemanager >Reporter: Miklos Szegedi >Priority: Minor > Labels: newbie > Attachments: YARN-7145.000.patch, YARN-7145.001.patch > > > I intend to add a 200 milliseconds sleep into AsyncDispatcher, and run the > job to identify the tests that are potentially flaky. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10100) [CS] Separate config validation steps from the update part
[ https://issues.apache.org/jira/browse/YARN-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton reassigned YARN-10100: --- Assignee: (was: Kinga Marton) > [CS] Separate config validation steps from the update part > -- > > Key: YARN-10100 > URL: https://issues.apache.org/jira/browse/YARN-10100 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Kinga Marton >Priority: Major > > In Capacity Scheduler initialisation/reinitialisation there are a lot of > validation steps performed. Some of this steps are deeply hidden. Having this > implementation is really hard to figure out what a valid configuration means. > Let's figure out what are exactly the validation steps and separate them from > the update part. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9883) Reshape SchedulerHealth class
[
https://issues.apache.org/jira/browse/YARN-9883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110001#comment-17110001
]
Kinga Marton commented on YARN-9883:
Yesy, [~BilwaST]. Feel free to assign it to yourself. Thanks
> Reshape SchedulerHealth class
> -
>
> Key: YARN-9883
> URL: https://issues.apache.org/jira/browse/YARN-9883
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Kinga Marton
>Priority: Minor
>
> The {{SchedulerHealth}} class has some flaws, for example:
> - It has no javadoc at all
> - All its objects are package-private: they should be private
> - The internal maps should be (Concurrent) EnumMaps instead of HashMaps: they
> are more efficient in storing Enums
> - schedulerHealthDetails only stores the last operation, its name should
> reflect that (just like lastSchedulerRunDetails)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-10148) Add Unit test for queue ACL for both FS and CS
[ https://issues.apache.org/jira/browse/YARN-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton reopened YARN-10148: - > Add Unit test for queue ACL for both FS and CS > -- > > Key: YARN-10148 > URL: https://issues.apache.org/jira/browse/YARN-10148 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-10148.001.patch, YARN-10148.002.patch, > YARN-10148.003.patch, YARN-10148.004.patch, YARN-10148.005.patch, > YARN-10148.006.patch, YARN-10148.amend001.patch > > > Add some unit tests covering the queue ACL evaluation for both FS and CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10148) Add Unit test for queue ACL for both FS and CS
[ https://issues.apache.org/jira/browse/YARN-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17047383#comment-17047383 ] Kinga Marton commented on YARN-10148: - Thank you [~snemeth] for the merge. I found a small bug in the merged patch, what is causing a test failure. I fixed it in [YARN-10148.amend001.patch|https://issues.apache.org/jira/secure/attachment/12994876/YARN-10148.amend001.patch]. Can you please check it? > Add Unit test for queue ACL for both FS and CS > -- > > Key: YARN-10148 > URL: https://issues.apache.org/jira/browse/YARN-10148 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-10148.001.patch, YARN-10148.002.patch, > YARN-10148.003.patch, YARN-10148.004.patch, YARN-10148.005.patch, > YARN-10148.006.patch, YARN-10148.amend001.patch > > > Add some unit tests covering the queue ACL evaluation for both FS and CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10148) Add Unit test for queue ACL for both FS and CS
[ https://issues.apache.org/jira/browse/YARN-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10148: Attachment: YARN-10148.amend001.patch > Add Unit test for queue ACL for both FS and CS > -- > > Key: YARN-10148 > URL: https://issues.apache.org/jira/browse/YARN-10148 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-10148.001.patch, YARN-10148.002.patch, > YARN-10148.003.patch, YARN-10148.004.patch, YARN-10148.005.patch, > YARN-10148.006.patch, YARN-10148.amend001.patch > > > Add some unit tests covering the queue ACL evaluation for both FS and CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10148) Add Unit test for queue ACL for both FS and CS
[ https://issues.apache.org/jira/browse/YARN-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10148: Attachment: YARN-10148.006.patch > Add Unit test for queue ACL for both FS and CS > -- > > Key: YARN-10148 > URL: https://issues.apache.org/jira/browse/YARN-10148 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10148.001.patch, YARN-10148.002.patch, > YARN-10148.003.patch, YARN-10148.004.patch, YARN-10148.005.patch, > YARN-10148.006.patch > > > Add some unit tests covering the queue ACL evaluation for both FS and CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10148) Add Unit test for queue ACL for both FS and CS
[ https://issues.apache.org/jira/browse/YARN-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046743#comment-17046743 ] Kinga Marton commented on YARN-10148: - I have attached a new patch with the reported of the checkstyle issues. > Add Unit test for queue ACL for both FS and CS > -- > > Key: YARN-10148 > URL: https://issues.apache.org/jira/browse/YARN-10148 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10148.001.patch, YARN-10148.002.patch, > YARN-10148.003.patch, YARN-10148.004.patch, YARN-10148.005.patch, > YARN-10148.006.patch > > > Add some unit tests covering the queue ACL evaluation for both FS and CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10148) Add Unit test for queue ACL for both FS and CS
[ https://issues.apache.org/jira/browse/YARN-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046557#comment-17046557 ] Kinga Marton commented on YARN-10148: - Thank you [~snemeth] for the review! I have addressed your comments in the newly attached patch > Add Unit test for queue ACL for both FS and CS > -- > > Key: YARN-10148 > URL: https://issues.apache.org/jira/browse/YARN-10148 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10148.001.patch, YARN-10148.002.patch, > YARN-10148.003.patch, YARN-10148.004.patch, YARN-10148.005.patch > > > Add some unit tests covering the queue ACL evaluation for both FS and CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10148) Add Unit test for queue ACL for both FS and CS
[ https://issues.apache.org/jira/browse/YARN-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10148: Attachment: YARN-10148.005.patch > Add Unit test for queue ACL for both FS and CS > -- > > Key: YARN-10148 > URL: https://issues.apache.org/jira/browse/YARN-10148 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10148.001.patch, YARN-10148.002.patch, > YARN-10148.003.patch, YARN-10148.004.patch, YARN-10148.005.patch > > > Add some unit tests covering the queue ACL evaluation for both FS and CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10167) FS-CS Converter: Need validate c-s.xml after converting
[ https://issues.apache.org/jira/browse/YARN-10167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046229#comment-17046229 ] Kinga Marton commented on YARN-10167: - There is a validation API implemented in YARN-10022. We can use that API to validate the generated configuration. > FS-CS Converter: Need validate c-s.xml after converting > --- > > Key: YARN-10167 > URL: https://issues.apache.org/jira/browse/YARN-10167 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Priority: Major > Labels: fs2cs, newbie > > Currently we just generated c-s.xml, but we didn't validate that. To make > sure the c-s.xml is correct after conversion, it's better to initialize the > CS scheduler using configs. > Also, in the test, we should try to leverage MockRM to validate generated > configs as much as we could. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10148) Add Unit test for queue ACL for both FS and CS
[ https://issues.apache.org/jira/browse/YARN-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10148: Attachment: YARN-10148.004.patch > Add Unit test for queue ACL for both FS and CS > -- > > Key: YARN-10148 > URL: https://issues.apache.org/jira/browse/YARN-10148 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10148.001.patch, YARN-10148.002.patch, > YARN-10148.003.patch, YARN-10148.004.patch > > > Add some unit tests covering the queue ACL evaluation for both FS and CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10148) Add Unit test for queue ACL for both FS and CS
[
https://issues.apache.org/jira/browse/YARN-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040871#comment-17040871
]
Kinga Marton commented on YARN-10148:
-
Thank you [~adam.antal] for the review. I have addressed your comments in
YARN-10148.003.patch:
- changed the newly introduced queue names to D and D1, where D1 is a child
queue of D. Also added a small comment for {{updateConfigWithDAndD1Queues}}
method
- {{in updateConfigWithDAndD1Queues}} there are mixed the queues from parent
class and the actual one because I just want to update the configuration with
the new queues instead of creating a new one. In this case if I would keep only
the new queues I would have to do a lot of configuration change related the old
ones (stop them, modify the capacities, etc.)
- fixed the indentation (mi idea was set to use 8 spaces for continuation
indent)
- Added assertion message
- reverted TestAbsoluteResourceConfiguration changes: yes, it was by mistake.
This was a good example of why I should not use QUEUE_C again :)
- replaced "test.build.data", in {{TestFairSchedulerQueueACLs with
GenericTestUtils.SYSPROP_TEST_DATA_DIR}}
> Add Unit test for queue ACL for both FS and CS
> --
>
> Key: YARN-10148
> URL: https://issues.apache.org/jira/browse/YARN-10148
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-10148.001.patch, YARN-10148.002.patch,
> YARN-10148.003.patch
>
>
> Add some unit tests covering the queue ACL evaluation for both FS and CS.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10148) Add Unit test for queue ACL for both FS and CS
[ https://issues.apache.org/jira/browse/YARN-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10148: Attachment: YARN-10148.003.patch > Add Unit test for queue ACL for both FS and CS > -- > > Key: YARN-10148 > URL: https://issues.apache.org/jira/browse/YARN-10148 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10148.001.patch, YARN-10148.002.patch, > YARN-10148.003.patch > > > Add some unit tests covering the queue ACL evaluation for both FS and CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10148) Add Unit test for queue ACL for both FS and CS
[ https://issues.apache.org/jira/browse/YARN-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040090#comment-17040090 ] Kinga Marton commented on YARN-10148: - The test failures are unrelated. > Add Unit test for queue ACL for both FS and CS > -- > > Key: YARN-10148 > URL: https://issues.apache.org/jira/browse/YARN-10148 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10148.001.patch, YARN-10148.002.patch > > > Add some unit tests covering the queue ACL evaluation for both FS and CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10143) YARN-10101 broke Yarn logs CLI
[
https://issues.apache.org/jira/browse/YARN-10143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039880#comment-17039880
]
Kinga Marton commented on YARN-10143:
-
Hi [~adam.antal], the patch LGTM, if you will test it with tFile as well and it
will work, than I give +1. (non-binding)
> YARN-10101 broke Yarn logs CLI
> ---
>
> Key: YARN-10143
> URL: https://issues.apache.org/jira/browse/YARN-10143
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.0, 3.2.2
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Blocker
> Attachments: YARN-10143.001.patch, YARN-10143.002.patch
>
>
> YARN-10101 broke the Yarn logs CLI.
> In {{LogsCLI#359}} a {{ContainerLogsRequest}} has been created with null set
> as appAttemptId, while the {{LogAggregationFileController}} instances are
> configured badly to handle this case. The new JHS API works as expected for
> defined application attempt, but we should fix this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10148) Add Unit test for queue ACL for both FS and CS
[ https://issues.apache.org/jira/browse/YARN-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10148: Attachment: YARN-10148.002.patch > Add Unit test for queue ACL for both FS and CS > -- > > Key: YARN-10148 > URL: https://issues.apache.org/jira/browse/YARN-10148 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10148.001.patch, YARN-10148.002.patch > > > Add some unit tests covering the queue ACL evaluation for both FS and CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10148) Add Unit test for queue ACL for both FS and CS
[ https://issues.apache.org/jira/browse/YARN-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10148: Attachment: YARN-10148.001.patch > Add Unit test for queue ACL for both FS and CS > -- > > Key: YARN-10148 > URL: https://issues.apache.org/jira/browse/YARN-10148 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10148.001.patch > > > Add some unit tests covering the queue ACL evaluation for both FS and CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10148) Add Unit test for queue ACL for both FS and CS
Kinga Marton created YARN-10148: --- Summary: Add Unit test for queue ACL for both FS and CS Key: YARN-10148 URL: https://issues.apache.org/jira/browse/YARN-10148 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Reporter: Kinga Marton Assignee: Kinga Marton Add some unit tests covering the queue ACL evaluation for both FS and CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10117) FS-CS converter: adjust queue ACL to have the same output with CS as for FS has
[ https://issues.apache.org/jira/browse/YARN-10117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton resolved YARN-10117. - Resolution: Not A Problem > FS-CS converter: adjust queue ACL to have the same output with CS as for FS > has > --- > > Key: YARN-10117 > URL: https://issues.apache.org/jira/browse/YARN-10117 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Kinga Marton >Priority: Major > > Both FS and CS seems to check the ACL recursively: from the leaf via the > parent(s) to the root (inclusive). However there are some differences in > evaluating them, what cause to have different results with the two schedulers. > Some examples are the following ones: > ||Tested scenario||FS output||CS output|| > |Root - “ ” > C - * > C1 - *|Denied by root ACL|OK| > |Root - “ ” > C - “ ” > C1 - “ ”|Denied by root ACL|OK| > |Root - “ ” > C - * > C1 - “ ”|Denied by root ACL|OK| > |Root - “ ” > C - “ ” > C1 - *|Denied by root ACL|OK| > Note: I have set the same value for both submit application and administer > queue ACLs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10117) FS-CS converter: adjust queue ACL to have the same output with CS as for FS has
[ https://issues.apache.org/jira/browse/YARN-10117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039028#comment-17039028 ] Kinga Marton commented on YARN-10117: - There is no need for adjustments, because I passed only the queue name instead of the full queue path, so the apps were placed in the wrong queue(root.c1 created dinamically instead of root.c.c1). Passing the full queue path will generate the same results. I will open a new Jira for creating some unit tests for the manually tested scenarios. > FS-CS converter: adjust queue ACL to have the same output with CS as for FS > has > --- > > Key: YARN-10117 > URL: https://issues.apache.org/jira/browse/YARN-10117 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Kinga Marton >Priority: Major > > Both FS and CS seems to check the ACL recursively: from the leaf via the > parent(s) to the root (inclusive). However there are some differences in > evaluating them, what cause to have different results with the two schedulers. > Some examples are the following ones: > ||Tested scenario||FS output||CS output|| > |Root - “ ” > C - * > C1 - *|Denied by root ACL|OK| > |Root - “ ” > C - “ ” > C1 - “ ”|Denied by root ACL|OK| > |Root - “ ” > C - * > C1 - “ ”|Denied by root ACL|OK| > |Root - “ ” > C - “ ” > C1 - *|Denied by root ACL|OK| > Note: I have set the same value for both submit application and administer > queue ACLs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10109) Allow stop and convert from leaf to parent queue in a single Mutation API call
[
https://issues.apache.org/jira/browse/YARN-10109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031615#comment-17031615
]
Kinga Marton commented on YARN-10109:
-
This approach looks safer. +1 from my side (non-binding)
> Allow stop and convert from leaf to parent queue in a single Mutation API call
> --
>
> Key: YARN-10109
> URL: https://issues.apache.org/jira/browse/YARN-10109
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-10109-001.patch, YARN-10109-002.patch,
> YARN-10109-003.patch, YARN-10109-004.patch
>
>
> SchedulerConf Mutation API does not Allow Stop and Adding queue under an
> existing Leaf Queue in a single call.
> *Repro:*
>
> {code:java}
> Capacity-Scheduler.xml:
> yarn.scheduler.capacity.root.queues = default
> yarn.scheduler.capacity.root.default.capacity = 100
> cat abc.xml
>
>
> root.default.v1
>
>
> capacity
> 100
>
>
>
>
> root.default
>
>
> state
> STOPPED
>
>
>
>
> [yarn@pjoseph-1 tmp]$ curl --negotiate -u : -X PUT -d @add.xml -H
> "Content-type: application/xml"
> 'http://:8088/ws/v1/cluster/scheduler-conf?user.name=yarn'
> Failed to re-init queues : Can not convert the leaf queue: root.default to
> parent queue since it is not yet in stopped state. Current State : RUNNING
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10117) FS-CS converter: adjust queue ACL to have the same output with CS as for FS has
Kinga Marton created YARN-10117: --- Summary: FS-CS converter: adjust queue ACL to have the same output with CS as for FS has Key: YARN-10117 URL: https://issues.apache.org/jira/browse/YARN-10117 Project: Hadoop YARN Issue Type: Sub-task Reporter: Kinga Marton Both FS and CS seems to check the ACL recursively: from the leaf via the parent(s) to the root (inclusive). However there are some differences in evaluating them, what cause to have different results with the two schedulers. Some examples are the following ones: ||Tested scenario||FS output||CS output|| |Root - “ ” C - * C1 - *|Denied by root ACL|OK| |Root - “ ” C - “ ” C1 - “ ”|Denied by root ACL|OK| |Root - “ ” C - * C1 - “ ”|Denied by root ACL|OK| |Root - “ ” C - “ ” C1 - *|Denied by root ACL|OK| Note: I have set the same value for both submit application and administer queue ACLs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029657#comment-17029657 ] Kinga Marton commented on YARN-10022: - Thank you, [~prabhujoseph]! On 3.1 branch the build seems to be broken. I can retrigger the precommit build to check it the hdfs team fixed it. > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-10022-branch-3.1.001.patch, > YARN-10022-branch-3.2.001.patch, YARN-10022.001.patch, YARN-10022.002.patch, > YARN-10022.003.patch, YARN-10022.004.patch, YARN-10022.005.patch, > YARN-10022.006.patch, YARN-10022.007.patch, YARN-10022.WIP.patch, > YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029205#comment-17029205 ] Kinga Marton commented on YARN-10022: - The compilation failure is unrelated and is caused by HDFS. I checked the tests as well and most of the failures are because the test case timed out. > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-10022-branch-3.1.001.patch, > YARN-10022-branch-3.2.001.patch, YARN-10022.001.patch, YARN-10022.002.patch, > YARN-10022.003.patch, YARN-10022.004.patch, YARN-10022.005.patch, > YARN-10022.006.patch, YARN-10022.007.patch, YARN-10022.WIP.patch, > YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10109) Allow stop and convert from leaf to parent queue in a single Mutation API call
[
https://issues.apache.org/jira/browse/YARN-10109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028787#comment-17028787
]
Kinga Marton commented on YARN-10109:
-
Thank you [~prabhujoseph]. LGTM, +1 (non-binding)
> Allow stop and convert from leaf to parent queue in a single Mutation API call
> --
>
> Key: YARN-10109
> URL: https://issues.apache.org/jira/browse/YARN-10109
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-10109-001.patch, YARN-10109-002.patch,
> YARN-10109-003.patch
>
>
> SchedulerConf Mutation API does not Allow Stop and Adding queue under an
> existing Leaf Queue in a single call.
> *Repro:*
>
> {code:java}
> Capacity-Scheduler.xml:
> yarn.scheduler.capacity.root.queues = default
> yarn.scheduler.capacity.root.default.capacity = 100
> cat abc.xml
>
>
> root.default.v1
>
>
> capacity
> 100
>
>
>
>
> root.default
>
>
> state
> STOPPED
>
>
>
>
> [yarn@pjoseph-1 tmp]$ curl --negotiate -u : -X PUT -d @add.xml -H
> "Content-type: application/xml"
> 'http://:8088/ws/v1/cluster/scheduler-conf?user.name=yarn'
> Failed to re-init queues : Can not convert the leaf queue: root.default to
> parent queue since it is not yet in stopped state. Current State : RUNNING
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10022: Attachment: YARN-10022-branch-3.1.001.patch > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-10022-branch-3.1.001.patch, > YARN-10022-branch-3.2.001.patch, YARN-10022.001.patch, YARN-10022.002.patch, > YARN-10022.003.patch, YARN-10022.004.patch, YARN-10022.005.patch, > YARN-10022.006.patch, YARN-10022.007.patch, YARN-10022.WIP.patch, > YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10022: Attachment: YARN-10022-branch-3.2.001.patch > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-10022-branch-3.2.001.patch, YARN-10022.001.patch, > YARN-10022.002.patch, YARN-10022.003.patch, YARN-10022.004.patch, > YARN-10022.005.patch, YARN-10022.006.patch, YARN-10022.007.patch, > YARN-10022.WIP.patch, YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9743) [JDK11] TestTimelineWebServices.testContextFactory fails
[
https://issues.apache.org/jira/browse/YARN-9743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025907#comment-17025907
]
Kinga Marton edited comment on YARN-9743 at 1/29/20 2:17 PM:
-
[~aajisaka] I have checked and tested your PR and it LGTM, +1 (non-binding).
Thank you!
was (Author: kmarton):
[~aajisaka] I have checked and tested your PR and it LGTM, +1 (non-binding)
> [JDK11] TestTimelineWebServices.testContextFactory fails
>
>
> Key: YARN-9743
> URL: https://issues.apache.org/jira/browse/YARN-9743
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineservice
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: YARN-9743.001.patch, YARN-9743.002.patch
>
>
> Tested on OpenJDK 11.0.2 on a Mac.
> Stack trace:
> {noformat}
> [ERROR] Tests run: 29, Failures: 0, Errors: 3, Skipped: 0, Time elapsed:
> 36.016 s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
> [ERROR]
> testContextFactory(org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices)
> Time elapsed: 1.031 s <<< ERROR!
> java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:315)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.newContext(ContextFactory.java:85)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.createContext(ContextFactory.java:112)
> at
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices.testContextFactory(TestTimelineWebServices.java:1039)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Commented] (YARN-9743) [JDK11] TestTimelineWebServices.testContextFactory fails
[
https://issues.apache.org/jira/browse/YARN-9743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025907#comment-17025907
]
Kinga Marton commented on YARN-9743:
[~aajisaka] I have checked and tested your PR and it LGTM, +1 (non-binding)
> [JDK11] TestTimelineWebServices.testContextFactory fails
>
>
> Key: YARN-9743
> URL: https://issues.apache.org/jira/browse/YARN-9743
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineservice
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: YARN-9743.001.patch, YARN-9743.002.patch
>
>
> Tested on OpenJDK 11.0.2 on a Mac.
> Stack trace:
> {noformat}
> [ERROR] Tests run: 29, Failures: 0, Errors: 3, Skipped: 0, Time elapsed:
> 36.016 s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
> [ERROR]
> testContextFactory(org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices)
> Time elapsed: 1.031 s <<< ERROR!
> java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:315)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.newContext(ContextFactory.java:85)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.createContext(ContextFactory.java:112)
> at
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices.testContextFactory(TestTimelineWebServices.java:1039)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9743) [JDK11] TestTimelineWebServices.testContextFactory fails
[
https://issues.apache.org/jira/browse/YARN-9743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025764#comment-17025764
]
Kinga Marton commented on YARN-9743:
Sorry, ignore my previous comment. I can see the precommit results in your PR.
> [JDK11] TestTimelineWebServices.testContextFactory fails
>
>
> Key: YARN-9743
> URL: https://issues.apache.org/jira/browse/YARN-9743
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineservice
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: YARN-9743.001.patch, YARN-9743.002.patch
>
>
> Tested on OpenJDK 11.0.2 on a Mac.
> Stack trace:
> {noformat}
> [ERROR] Tests run: 29, Failures: 0, Errors: 3, Skipped: 0, Time elapsed:
> 36.016 s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
> [ERROR]
> testContextFactory(org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices)
> Time elapsed: 1.031 s <<< ERROR!
> java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:315)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.newContext(ContextFactory.java:85)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.createContext(ContextFactory.java:112)
> at
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices.testContextFactory(TestTimelineWebServices.java:1039)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10109) Allow stop and convert from leaf to parent queue in a single Mutation API call
[
https://issues.apache.org/jira/browse/YARN-10109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025760#comment-17025760
]
Kinga Marton commented on YARN-10109:
-
Thank you for reporting fixing this [~prabhujoseph].
I would have a small comment to the following code:
{code:java}
try {
newQueueState = QueueState.valueOf(
newConf.get(configPrefix + "state"));
} catch (Exception ex) {
// ignore the exception as the config state is optional133
}
{code}
It is never a good idea to just suppress/ignore an Exception. I think at least
we should log a message.
> Allow stop and convert from leaf to parent queue in a single Mutation API call
> --
>
> Key: YARN-10109
> URL: https://issues.apache.org/jira/browse/YARN-10109
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-10109-001.patch, YARN-10109-002.patch
>
>
> SchedulerConf Mutation API does not Allow Stop and Adding queue under an
> existing Leaf Queue in a single call.
> *Repro:*
>
> {code:java}
> Capacity-Scheduler.xml:
> yarn.scheduler.capacity.root.queues = default
> yarn.scheduler.capacity.root.default.capacity = 100
> cat abc.xml
>
>
> root.default.v1
>
>
> capacity
> 100
>
>
>
>
> root.default
>
>
> state
> STOPPED
>
>
>
>
> [yarn@pjoseph-1 tmp]$ curl --negotiate -u : -X PUT -d @add.xml -H
> "Content-type: application/xml"
> 'http://:8088/ws/v1/cluster/scheduler-conf?user.name=yarn'
> Failed to re-init queues : Can not convert the leaf queue: root.default to
> parent queue since it is not yet in stopped state. Current State : RUNNING
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9743) [JDK11] TestTimelineWebServices.testContextFactory fails
[
https://issues.apache.org/jira/browse/YARN-9743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025702#comment-17025702
]
Kinga Marton commented on YARN-9743:
Thank you [~aajisaka] for taking this over. Can you please upload your patch
here, since we are not moved yet to pull request and the precommit check can
pick only the patch files attached.
> [JDK11] TestTimelineWebServices.testContextFactory fails
>
>
> Key: YARN-9743
> URL: https://issues.apache.org/jira/browse/YARN-9743
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineservice
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: YARN-9743.001.patch, YARN-9743.002.patch
>
>
> Tested on OpenJDK 11.0.2 on a Mac.
> Stack trace:
> {noformat}
> [ERROR] Tests run: 29, Failures: 0, Errors: 3, Skipped: 0, Time elapsed:
> 36.016 s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
> [ERROR]
> testContextFactory(org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices)
> Time elapsed: 1.031 s <<< ERROR!
> java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:315)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.newContext(ContextFactory.java:85)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.createContext(ContextFactory.java:112)
> at
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices.testContextFactory(TestTimelineWebServices.java:1039)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9743) [JDK11] TestTimelineWebServices.testContextFactory fails
[
https://issues.apache.org/jira/browse/YARN-9743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024963#comment-17024963
]
Kinga Marton edited comment on YARN-9743 at 1/28/20 3:09 PM:
-
Hi [~aajisaka]. Unfortunately I did not had time to continue the work/to finish
this because of other priorities.
was (Author: kmarton):
Hi [~aajisaka]. Unfortunately I did not had time to continue the work on this
because of other priorities.
> [JDK11] TestTimelineWebServices.testContextFactory fails
>
>
> Key: YARN-9743
> URL: https://issues.apache.org/jira/browse/YARN-9743
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineservice
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-9743.001.patch, YARN-9743.002.patch
>
>
> Tested on OpenJDK 11.0.2 on a Mac.
> Stack trace:
> {noformat}
> [ERROR] Tests run: 29, Failures: 0, Errors: 3, Skipped: 0, Time elapsed:
> 36.016 s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
> [ERROR]
> testContextFactory(org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices)
> Time elapsed: 1.031 s <<< ERROR!
> java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:315)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.newContext(ContextFactory.java:85)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.createContext(ContextFactory.java:112)
> at
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices.testContextFactory(TestTimelineWebServices.java:1039)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For
[jira] [Commented] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[
https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025174#comment-17025174
]
Kinga Marton commented on YARN-10022:
-
The failure is unrelated: actually it timed out:
{{Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M1:test (default-test) on
project hadoop-yarn-server-resourcemanager: There was a timeout or other error
in the fork}}
However I will retrigger the build to have a clean one.
> Create RM Rest API to validate a CapacityScheduler Configuration
>
>
> Key: YARN-10022
> URL: https://issues.apache.org/jira/browse/YARN-10022
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-10022.001.patch, YARN-10022.002.patch,
> YARN-10022.003.patch, YARN-10022.004.patch, YARN-10022.005.patch,
> YARN-10022.006.patch, YARN-10022.007.patch, YARN-10022.WIP.patch,
> YARN-10022.WIP2.patch
>
>
> RMWebService should expose a new api which gets a CapacityScheduler
> Configuration as an input, validates it and returns success / failure.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10022: Attachment: YARN-10022.007.patch > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.001.patch, YARN-10022.002.patch, > YARN-10022.003.patch, YARN-10022.004.patch, YARN-10022.005.patch, > YARN-10022.006.patch, YARN-10022.007.patch, YARN-10022.WIP.patch, > YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9743) [JDK11] TestTimelineWebServices.testContextFactory fails
[
https://issues.apache.org/jira/browse/YARN-9743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024963#comment-17024963
]
Kinga Marton commented on YARN-9743:
Hi [~aajisaka]. Unfortunately I did not had time to continue the work on this
because of other priorities.
> [JDK11] TestTimelineWebServices.testContextFactory fails
>
>
> Key: YARN-9743
> URL: https://issues.apache.org/jira/browse/YARN-9743
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineservice
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-9743.001.patch, YARN-9743.002.patch
>
>
> Tested on OpenJDK 11.0.2 on a Mac.
> Stack trace:
> {noformat}
> [ERROR] Tests run: 29, Failures: 0, Errors: 3, Skipped: 0, Time elapsed:
> 36.016 s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
> [ERROR]
> testContextFactory(org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices)
> Time elapsed: 1.031 s <<< ERROR!
> java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:315)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.newContext(ContextFactory.java:85)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.createContext(ContextFactory.java:112)
> at
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices.testContextFactory(TestTimelineWebServices.java:1039)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024962#comment-17024962 ] Kinga Marton commented on YARN-10022: - Thank you [~prabhujoseph]! I have addressed your findings in the new patch. A small comment: "_testValidateMemoryAllocation needs assert_" and "_testValidateVCores needs assert_" - for this test cases I have added a comment because there is nothing we can assert, since the tested method does not call other methods, and it is not returning anything. > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.001.patch, YARN-10022.002.patch, > YARN-10022.003.patch, YARN-10022.004.patch, YARN-10022.005.patch, > YARN-10022.006.patch, YARN-10022.WIP.patch, YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10022: Attachment: YARN-10022.006.patch > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.001.patch, YARN-10022.002.patch, > YARN-10022.003.patch, YARN-10022.004.patch, YARN-10022.005.patch, > YARN-10022.006.patch, YARN-10022.WIP.patch, YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024405#comment-17024405 ] Kinga Marton commented on YARN-10022: - Thank you [~prabhujoseph] for fixing the issue. I have uploaded a new patch what contains the request test cases as well. [~prabhujoseph], [~snemeth] can you please check the latest patch? > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.001.patch, YARN-10022.002.patch, > YARN-10022.003.patch, YARN-10022.004.patch, YARN-10022.005.patch, > YARN-10022.WIP.patch, YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10022: Attachment: YARN-10022.005.patch > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.001.patch, YARN-10022.002.patch, > YARN-10022.003.patch, YARN-10022.004.patch, YARN-10022.005.patch, > YARN-10022.WIP.patch, YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10022: Attachment: YARN-10022.002.patch > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.001.patch, YARN-10022.002.patch, > YARN-10022.WIP.patch, YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021963#comment-17021963 ] Kinga Marton commented on YARN-10022: - Thank you [~prabhujoseph] for checking the patch. I have uploaded a new one with the fixes + added some unit tests. I have also opened a follow up issue (YARN-10100) for separating the validation part from the initialisation part. > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.001.patch, YARN-10022.WIP.patch, > YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10100) [CS] Separate config validation steps from the update part
Kinga Marton created YARN-10100: --- Summary: [CS] Separate config validation steps from the update part Key: YARN-10100 URL: https://issues.apache.org/jira/browse/YARN-10100 Project: Hadoop YARN Issue Type: Improvement Components: capacity scheduler Reporter: Kinga Marton Assignee: Kinga Marton In Capacity Scheduler initialisation/reinitialisation there are a lot of validation steps performed. Some of this steps are deeply hidden. Having this implementation is really hard to figure out what a valid configuration means. Let's figure out what are exactly the validation steps and separate them from the update part. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10022: Attachment: YARN-10022.001.patch > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.001.patch, YARN-10022.WIP.patch, > YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016855#comment-17016855
]
Kinga Marton commented on YARN-10070:
-
hank you [~prabhujoseph] and [~adam.antal] for the review!
> NPE if no rule is defined and application-tag-based-placement is enabled
>
>
> Key: YARN-10070
> URL: https://issues.apache.org/jira/browse/YARN-10070
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-10070.001.patch, YARN-10070.002.patch,
> YARN-10070.003.patch
>
>
> If there is no rule defined for a user NPE is thrown by the following line.
> {code:java}
> String queue = placementManager
> .placeApplication(context, usernameUsedForPlacement).getQueue();{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016855#comment-17016855
]
Kinga Marton edited comment on YARN-10070 at 1/16/20 12:22 PM:
---
Thank you [~prabhujoseph] and [~adam.antal] for the review!
was (Author: kmarton):
hank you [~prabhujoseph] and [~adam.antal] for the review!
> NPE if no rule is defined and application-tag-based-placement is enabled
>
>
> Key: YARN-10070
> URL: https://issues.apache.org/jira/browse/YARN-10070
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-10070.001.patch, YARN-10070.002.patch,
> YARN-10070.003.patch
>
>
> If there is no rule defined for a user NPE is thrown by the following line.
> {code:java}
> String queue = placementManager
> .placeApplication(context, usernameUsedForPlacement).getQueue();{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10022: Attachment: YARN-10022.WIP2.patch > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.WIP.patch, YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015937#comment-17015937
]
Kinga Marton commented on YARN-10070:
-
Thank you [~adam.antal] for the review. As we have discussed offline, I have
changed the Javadoc comment for the test cases from "_Test case for when..._"
to "_Test for the case when..._"
> NPE if no rule is defined and application-tag-based-placement is enabled
>
>
> Key: YARN-10070
> URL: https://issues.apache.org/jira/browse/YARN-10070
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-10070.001.patch, YARN-10070.002.patch,
> YARN-10070.003.patch
>
>
> If there is no rule defined for a user NPE is thrown by the following line.
> {code:java}
> String queue = placementManager
> .placeApplication(context, usernameUsedForPlacement).getQueue();{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kinga Marton updated YARN-10070:
Attachment: YARN-10070.003.patch
> NPE if no rule is defined and application-tag-based-placement is enabled
>
>
> Key: YARN-10070
> URL: https://issues.apache.org/jira/browse/YARN-10070
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-10070.001.patch, YARN-10070.002.patch,
> YARN-10070.003.patch
>
>
> If there is no rule defined for a user NPE is thrown by the following line.
> {code:java}
> String queue = placementManager
> .placeApplication(context, usernameUsedForPlacement).getQueue();{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015922#comment-17015922 ] Kinga Marton commented on YARN-10022: - The attached patch is a Work in progress one, because I have to cover the changes with some test cases, and also I expect to have some checkstyle issues and some failing tests. Some comments related to the patch: * we need an API to decide whether a capacity scheduler configuration change is valid or not ad return the new merged configuration if it is a valid one. * my first idea was to extract all the verification steps fro the actual code and write a clear verification part. Unfortunately I couldn't extract all the steps, so at one point I need to create a new instance of CS and call reinitialise queues to be sure that all the validation steps are done. For this part I will open a follow up Jira to try to extract that part as well and remove that hacky part. * I have added a test file, what contains only the signature of the test cases I want to implement, any other test case ideas are very welcomed. :) > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.WIP.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10022: Attachment: (was: YARN-YARN-10022.001.patch) > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.WIP.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10022: Attachment: YARN-10022.WIP.patch > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.WIP.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10022: Attachment: YARN-YARN-10022.001.patch > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-YARN-10022.001.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015012#comment-17015012
]
Kinga Marton edited comment on YARN-10070 at 1/14/20 11:19 AM:
---
Thank you [~prabhujoseph] for the review! I have uploaded a new patch with
fixing the checkstyle issues.
was (Author: kmarton):
Thank you [~prabhujoseph] for the review! I have uploaded a new patch with the
checkstyle issues.
> NPE if no rule is defined and application-tag-based-placement is enabled
>
>
> Key: YARN-10070
> URL: https://issues.apache.org/jira/browse/YARN-10070
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-10070.001.patch, YARN-10070.002.patch
>
>
> If there is no rule defined for a user NPE is thrown by the following line.
> {code:java}
> String queue = placementManager
> .placeApplication(context, usernameUsedForPlacement).getQueue();{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kinga Marton updated YARN-10070:
Attachment: YARN-10070.002.patch
> NPE if no rule is defined and application-tag-based-placement is enabled
>
>
> Key: YARN-10070
> URL: https://issues.apache.org/jira/browse/YARN-10070
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-10070.001.patch, YARN-10070.002.patch
>
>
> If there is no rule defined for a user NPE is thrown by the following line.
> {code:java}
> String queue = placementManager
> .placeApplication(context, usernameUsedForPlacement).getQueue();{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015012#comment-17015012
]
Kinga Marton commented on YARN-10070:
-
Thank you [~prabhujoseph] for the review! I have uploaded a new patch with the
checkstyle issues.
> NPE if no rule is defined and application-tag-based-placement is enabled
>
>
> Key: YARN-10070
> URL: https://issues.apache.org/jira/browse/YARN-10070
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-10070.001.patch, YARN-10070.002.patch
>
>
> If there is no rule defined for a user NPE is thrown by the following line.
> {code:java}
> String queue = placementManager
> .placeApplication(context, usernameUsedForPlacement).getQueue();{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kinga Marton updated YARN-10070:
Attachment: YARN-10070.001.patch
> NPE if no rule is defined and application-tag-based-placement is enabled
>
>
> Key: YARN-10070
> URL: https://issues.apache.org/jira/browse/YARN-10070
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-10070.001.patch
>
>
> If there is no rule defined for a user NPE is thrown by the following line.
> {code:java}
> String queue = placementManager
> .placeApplication(context, usernameUsedForPlacement).getQueue();{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013847#comment-17013847
]
Kinga Marton edited comment on YARN-10070 at 1/12/20 6:44 PM:
--
Thank you [~prabhujoseph] for checking it.
I think I found the bug in the code part you have shared:
* the logic it should be followed is: we should check if the submitting user
(hive) has access to submit application in the queue of the user from the
application tag (tezq). If no, the placement will be done using the submitting
user (hive) instead of the one from the application tag (ambart-qa), so it
should be placed to the default queue.
* The bug is in the following code:
{code:java}
String queue = placementManager
.placeApplication(context, usernameUsedForPlacement).getQueue();
{code}
- here {{usernameUsedForPlacement}} should be replaced with
{{userNameFromApptag}} in order to get the queue the application would be
placed.
{code:java}
UserGroupInformation callerUGI = UserGroupInformation
.createRemoteUser(userNameFromAppTag);
{code}
- here {{userNameFromAppTag}} should be replaced with {{user}}, so we can
check than if the user(hive) has access to submit applications to the queue
where the application would be placed using the username from the apptag(tezq).
was (Author: kmarton):
Thank you [~prabhujoseph] for checking it.
I think I found the bug in the code part you have shared:
* the logic it should be followed is: we should check if the submitting use
(hive) has access to submit application in the queue of the user from the
application tag (tezq). If no, the placement will be done using the submitting
user (hive) instead of the one from the application tag (ambart-qa), so it
should be placed to the default queue.
* The bug is in the following code:
{code:java}
String queue = placementManager
.placeApplication(context, usernameUsedForPlacement).getQueue();
{code}
- here {{usernameUsedForPlacement}} should be replaces with
{{userNameFromApptag}} in order to get the queue the application would be
placed.
{code:java}
UserGroupInformation callerUGI = UserGroupInformation
.createRemoteUser(userNameFromAppTag);
{code}
- here {{userNameFromAppTag}} should be replaced with {{user}}, so we can check
than if the user(hive) has access to submit applications to the queue where the
application would be placed using the username from the apptag(tezq).
> NPE if no rule is defined and application-tag-based-placement is enabled
>
>
> Key: YARN-10070
> URL: https://issues.apache.org/jira/browse/YARN-10070
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
>
> If there is no rule defined for a user NPE is thrown by the following line.
> {code:java}
> String queue = placementManager
> .placeApplication(context, usernameUsedForPlacement).getQueue();{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013847#comment-17013847
]
Kinga Marton commented on YARN-10070:
-
Thank you [~prabhujoseph] for checking it.
I think I found the bug in the code part you have shared:
* the logic it should be followed is: we should check if the submitting use
(hive) has access to submit application in the queue of the user from the
application tag (tezq). If no, the placement will be done using the submitting
user (hive) instead of the one from the application tag (ambart-qa), so it
should be placed to the default queue.
* The bug is in the following code:
{code:java}
String queue = placementManager
.placeApplication(context, usernameUsedForPlacement).getQueue();
{code}
- here {{usernameUsedForPlacement}} should be replaces with
{{userNameFromApptag}} in order to get the queue the application would be
placed.
{code:java}
UserGroupInformation callerUGI = UserGroupInformation
.createRemoteUser(userNameFromAppTag);
{code}
- here {{userNameFromAppTag}} should be replaced with {{user}}, so we can check
than if the user(hive) has access to submit applications to the queue where the
application would be placed using the username from the apptag(tezq).
> NPE if no rule is defined and application-tag-based-placement is enabled
>
>
> Key: YARN-10070
> URL: https://issues.apache.org/jira/browse/YARN-10070
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
>
> If there is no rule defined for a user NPE is thrown by the following line.
> {code:java}
> String queue = placementManager
> .placeApplication(context, usernameUsedForPlacement).getQueue();{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
Kinga Marton created YARN-10070:
---
Summary: NPE if no rule is defined and
application-tag-based-placement is enabled
Key: YARN-10070
URL: https://issues.apache.org/jira/browse/YARN-10070
Project: Hadoop YARN
Issue Type: Bug
Reporter: Kinga Marton
Assignee: Kinga Marton
If there is no rule defined for a user NPE is thrown by the following line.
{code:java}
String queue = placementManager
.placeApplication(context, usernameUsedForPlacement).getQueue();{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
Kinga Marton created YARN-10022: --- Summary: Create RM Rest API to validate a CapacityScheduler Configuration Key: YARN-10022 URL: https://issues.apache.org/jira/browse/YARN-10022 Project: Hadoop YARN Issue Type: New Feature Reporter: Kinga Marton Assignee: Kinga Marton RMWebService should expose a new api which gets a CapacityScheduler Configuration as an input, validates it and returns success / failure. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9883) Reshape SchedulerHealth class
[
https://issues.apache.org/jira/browse/YARN-9883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991656#comment-16991656
]
Kinga Marton commented on YARN-9883:
[~adam.antal], unfortunately I didn't had time to start the work on this. So I
can wait with this until your changes will be merged.
> Reshape SchedulerHealth class
> -
>
> Key: YARN-9883
> URL: https://issues.apache.org/jira/browse/YARN-9883
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Kinga Marton
>Priority: Minor
>
> The {{SchedulerHealth}} class has some flaws, for example:
> - It has no javadoc at all
> - All its objects are package-private: they should be private
> - The internal maps should be (Concurrent) EnumMaps instead of HashMaps: they
> are more efficient in storing Enums
> - schedulerHealthDetails only stores the last operation, its name should
> reflect that (just like lastSchedulerRunDetails)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9886) Queue mapping based on userid passed through application tag
[ https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-9886: --- Attachment: YARN-9886.004.patch > Queue mapping based on userid passed through application tag > > > Key: YARN-9886 > URL: https://issues.apache.org/jira/browse/YARN-9886 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-9886-WIP.patch, YARN-9886.001.patch, > YARN-9886.002.patch, YARN-9886.003.patch, YARN-9886.004.patch > > > There are situations when the real submitting user differs from the user what > arrives to YARN. For example in case of a Hive application when Hive > impersonation is turned off, the hive queries will run as Hive user and the > mapping is done based on this username. Unfortunately in this case YARN > doesn't have any information about the real user and there are cases when the > customer may want to map these applications to the real submitting user's > queue instead of the Hive queue. > For these cases, if they would pass the username in the application tag we > may read it and use it during the queue mapping, if that user has rights to > run on the real user's queue. > [~sunilg] please correct me if I missed something. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9886) Queue mapping based on userid passed through application tag
[ https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-9886: --- Attachment: YARN-9886.003.patch > Queue mapping based on userid passed through application tag > > > Key: YARN-9886 > URL: https://issues.apache.org/jira/browse/YARN-9886 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-9886-WIP.patch, YARN-9886.001.patch, > YARN-9886.002.patch, YARN-9886.003.patch > > > There are situations when the real submitting user differs from the user what > arrives to YARN. For example in case of a Hive application when Hive > impersonation is turned off, the hive queries will run as Hive user and the > mapping is done based on this username. Unfortunately in this case YARN > doesn't have any information about the real user and there are cases when the > customer may want to map these applications to the real submitting user's > queue instead of the Hive queue. > For these cases, if they would pass the username in the application tag we > may read it and use it during the queue mapping, if that user has rights to > run on the real user's queue. > [~sunilg] please correct me if I missed something. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9886) Queue mapping based on userid passed through application tag
[ https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-9886: --- Attachment: YARN-9886.002.patch > Queue mapping based on userid passed through application tag > > > Key: YARN-9886 > URL: https://issues.apache.org/jira/browse/YARN-9886 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-9886-WIP.patch, YARN-9886.001.patch, > YARN-9886.002.patch > > > There are situations when the real submitting user differs from the user what > arrives to YARN. For example in case of a Hive application when Hive > impersonation is turned off, the hive queries will run as Hive user and the > mapping is done based on this username. Unfortunately in this case YARN > doesn't have any information about the real user and there are cases when the > customer may want to map these applications to the real submitting user's > queue instead of the Hive queue. > For these cases, if they would pass the username in the application tag we > may read it and use it during the queue mapping, if that user has rights to > run on the real user's queue. > [~sunilg] please correct me if I missed something. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9890) [UI2] Add Application tag to the app table and app detail page.
[ https://issues.apache.org/jira/browse/YARN-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969354#comment-16969354 ] Kinga Marton commented on YARN-9890: Thank you [~snemeth] for checking this parch! The new column is not displayed in the app attempts table. > [UI2] Add Application tag to the app table and app detail page. > --- > > Key: YARN-9890 > URL: https://issues.apache.org/jira/browse/YARN-9890 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: UI2_ApplicationTag.png, YARN-9890.001.patch > > > Right now AFAIK there is no possibility to filter the applications based on > the application tag in the UI. Adding this new column to the app table will > make this filtering possible as well. > From the UI2 this information is missing from the application detail page as > well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9743) [JDK11] TestTimelineWebServices.testContextFactory fails
[
https://issues.apache.org/jira/browse/YARN-9743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kinga Marton updated YARN-9743:
---
Attachment: YARN-9743.002.patch
> [JDK11] TestTimelineWebServices.testContextFactory fails
>
>
> Key: YARN-9743
> URL: https://issues.apache.org/jira/browse/YARN-9743
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineservice
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-9743.001.patch, YARN-9743.002.patch
>
>
> Tested on OpenJDK 11.0.2 on a Mac.
> Stack trace:
> {noformat}
> [ERROR] Tests run: 29, Failures: 0, Errors: 3, Skipped: 0, Time elapsed:
> 36.016 s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
> [ERROR]
> testContextFactory(org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices)
> Time elapsed: 1.031 s <<< ERROR!
> java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:315)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.newContext(ContextFactory.java:85)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.createContext(ContextFactory.java:112)
> at
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices.testContextFactory(TestTimelineWebServices.java:1039)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9743) [JDK11] TestTimelineWebServices.testContextFactory fails
[
https://issues.apache.org/jira/browse/YARN-9743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959643#comment-16959643
]
Kinga Marton commented on YARN-9743:
I think that the test failure is related to the fix, so I will have to fix it.
The install error is because of different versions of
{{javax.xml.bind:jaxb-api}}.
> [JDK11] TestTimelineWebServices.testContextFactory fails
>
>
> Key: YARN-9743
> URL: https://issues.apache.org/jira/browse/YARN-9743
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineservice
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-9743.001.patch
>
>
> Tested on OpenJDK 11.0.2 on a Mac.
> Stack trace:
> {noformat}
> [ERROR] Tests run: 29, Failures: 0, Errors: 3, Skipped: 0, Time elapsed:
> 36.016 s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
> [ERROR]
> testContextFactory(org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices)
> Time elapsed: 1.031 s <<< ERROR!
> java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:315)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.newContext(ContextFactory.java:85)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.createContext(ContextFactory.java:112)
> at
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices.testContextFactory(TestTimelineWebServices.java:1039)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9743) [JDK11] TestTimelineWebServices.testContextFactory fails
[
https://issues.apache.org/jira/browse/YARN-9743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kinga Marton updated YARN-9743:
---
Attachment: YARN-9743.001.patch
> [JDK11] TestTimelineWebServices.testContextFactory fails
>
>
> Key: YARN-9743
> URL: https://issues.apache.org/jira/browse/YARN-9743
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineservice
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-9743.001.patch
>
>
> Tested on OpenJDK 11.0.2 on a Mac.
> Stack trace:
> {noformat}
> [ERROR] Tests run: 29, Failures: 0, Errors: 3, Skipped: 0, Time elapsed:
> 36.016 s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
> [ERROR]
> testContextFactory(org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices)
> Time elapsed: 1.031 s <<< ERROR!
> java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:315)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.newContext(ContextFactory.java:85)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.createContext(ContextFactory.java:112)
> at
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices.testContextFactory(TestTimelineWebServices.java:1039)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9886) Queue mapping based on userid passed through application tag
[
https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956938#comment-16956938
]
Kinga Marton commented on YARN-9886:
In the attached patch 001 I have addressed the following issues:
* changed the id pattern from {{userid=}} to {{u=}}
* added property for enabling this feature
* added a property for specifying the users who can do such operation
* added unit tests
I also wanted to add a small information to the documentation, but I didn't
found the proper place for it. I was searching for a part where the scheduler
common things are documented.
> Queue mapping based on userid passed through application tag
>
>
> Key: YARN-9886
> URL: https://issues.apache.org/jira/browse/YARN-9886
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-9886-WIP.patch, YARN-9886.001.patch
>
>
> There are situations when the real submitting user differs from the user what
> arrives to YARN. For example in case of a Hive application when Hive
> impersonation is turned off, the hive queries will run as Hive user and the
> mapping is done based on this username. Unfortunately in this case YARN
> doesn't have any information about the real user and there are cases when the
> customer may want to map this applications to the real submitting user's
> queue instead of the Hive one.
> For this cases if they would pass the username in the application tag we may
> read it and use that one during the queue mapping, if that user has rights to
> run on the real user's queue.
> [~sunilg] please correct me if I missed something.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9886) Queue mapping based on userid passed through application tag
[ https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-9886: --- Attachment: YARN-9886.001.patch > Queue mapping based on userid passed through application tag > > > Key: YARN-9886 > URL: https://issues.apache.org/jira/browse/YARN-9886 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-9886-WIP.patch, YARN-9886.001.patch > > > There are situations when the real submitting user differs from the user what > arrives to YARN. For example in case of a Hive application when Hive > impersonation is turned off, the hive queries will run as Hive user and the > mapping is done based on this username. Unfortunately in this case YARN > doesn't have any information about the real user and there are cases when the > customer may want to map this applications to the real submitting user's > queue instead of the Hive one. > For this cases if they would pass the username in the application tag we may > read it and use that one during the queue mapping, if that user has rights to > run on the real user's queue. > [~sunilg] please correct me if I missed something. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9886) Queue mapping based on userid passed through application tag
[ https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956730#comment-16956730 ] Kinga Marton edited comment on YARN-9886 at 10/22/19 7:41 AM: -- [~wangda] yes. I will add a whitelist, where it can be defined who can use this feature. was (Author: kmarton): [~wangda] yes. I will add whitelist, where it can be defined who can use this feature. > Queue mapping based on userid passed through application tag > > > Key: YARN-9886 > URL: https://issues.apache.org/jira/browse/YARN-9886 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-9886-WIP.patch > > > There are situations when the real submitting user differs from the user what > arrives to YARN. For example in case of a Hive application when Hive > impersonation is turned off, the hive queries will run as Hive user and the > mapping is done based on this username. Unfortunately in this case YARN > doesn't have any information about the real user and there are cases when the > customer may want to map this applications to the real submitting user's > queue instead of the Hive one. > For this cases if they would pass the username in the application tag we may > read it and use that one during the queue mapping, if that user has rights to > run on the real user's queue. > [~sunilg] please correct me if I missed something. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9886) Queue mapping based on userid passed through application tag
[ https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956730#comment-16956730 ] Kinga Marton commented on YARN-9886: [~wangda] yes. I will add whitelist, where it can be defined who can use this feature. > Queue mapping based on userid passed through application tag > > > Key: YARN-9886 > URL: https://issues.apache.org/jira/browse/YARN-9886 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-9886-WIP.patch > > > There are situations when the real submitting user differs from the user what > arrives to YARN. For example in case of a Hive application when Hive > impersonation is turned off, the hive queries will run as Hive user and the > mapping is done based on this username. Unfortunately in this case YARN > doesn't have any information about the real user and there are cases when the > customer may want to map this applications to the real submitting user's > queue instead of the Hive one. > For this cases if they would pass the username in the application tag we may > read it and use that one during the queue mapping, if that user has rights to > run on the real user's queue. > [~sunilg] please correct me if I missed something. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9886) Queue mapping based on userid passed through application tag
[
https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955920#comment-16955920
]
Kinga Marton edited comment on YARN-9886 at 10/21/19 1:58 PM:
--
As a conclusion of an offline discussion with [~sunilg] we have agreed on the
following:
* We will keep the username in the tag, but I will change the pattern to
{{u=}}
* I will add a configuration to enable/disable the application placement using
the application tag. This way we can keep the backward compatibility and avoid
any undesired side effects.
was (Author: kmarton):
As a conclusion of an offline discussion with [~sunilg] we have agreed on the
following:
* We will keep the username in the tag, but I will change the pattern to
{{u=}}
* I will add a configuration to enable/disable the application placement using
the application tag. This way we can keep the backport compatibility and avoid
any undesired side effects.
> Queue mapping based on userid passed through application tag
>
>
> Key: YARN-9886
> URL: https://issues.apache.org/jira/browse/YARN-9886
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-9886-WIP.patch
>
>
> There are situations when the real submitting user differs from the user what
> arrives to YARN. For example in case of a Hive application when Hive
> impersonation is turned off, the hive queries will run as Hive user and the
> mapping is done based on this username. Unfortunately in this case YARN
> doesn't have any information about the real user and there are cases when the
> customer may want to map this applications to the real submitting user's
> queue instead of the Hive one.
> For this cases if they would pass the username in the application tag we may
> read it and use that one during the queue mapping, if that user has rights to
> run on the real user's queue.
> [~sunilg] please correct me if I missed something.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9886) Queue mapping based on userid passed through application tag
[
https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955920#comment-16955920
]
Kinga Marton commented on YARN-9886:
As a conclusion of an offline discussion with [~sunilg] we have agreed on the
following:
* We will keep the username in the tag, but I will change the pattern to
{{u=}}
* I will add a configuration to enable/disable the application placement using
the application tag. This way we can keep the backport compatibility and avoid
any undesired side effects.
> Queue mapping based on userid passed through application tag
>
>
> Key: YARN-9886
> URL: https://issues.apache.org/jira/browse/YARN-9886
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-9886-WIP.patch
>
>
> There are situations when the real submitting user differs from the user what
> arrives to YARN. For example in case of a Hive application when Hive
> impersonation is turned off, the hive queries will run as Hive user and the
> mapping is done based on this username. Unfortunately in this case YARN
> doesn't have any information about the real user and there are cases when the
> customer may want to map this applications to the real submitting user's
> queue instead of the Hive one.
> For this cases if they would pass the username in the application tag we may
> read it and use that one during the queue mapping, if that user has rights to
> run on the real user's queue.
> [~sunilg] please correct me if I missed something.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9886) Queue mapping based on userid passed through application tag
[ https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952736#comment-16952736 ] Kinga Marton edited comment on YARN-9886 at 10/16/19 2:16 PM: -- Sure, [~leftnoteasy]. I have converted them to separate issues. was (Author: kmarton): Sure, [~leftnoteasy]. I have converted them to separate issue. > Queue mapping based on userid passed through application tag > > > Key: YARN-9886 > URL: https://issues.apache.org/jira/browse/YARN-9886 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-9886-WIP.patch > > > There are situations when the real submitting user differs from the user what > arrives to YARN. For example in case of a Hive application when Hive > impersonation is turned off, the hive queries will run as Hive user and the > mapping is done based on this username. Unfortunately in this case YARN > doesn't have any information about the real user and there are cases when the > customer may want to map this applications to the real submitting user's > queue instead of the Hive one. > For this cases if they would pass the username in the application tag we may > read it and use that one during the queue mapping, if that user has rights to > run on the real user's queue. > [~sunilg] please correct me if I missed something. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9889) [UI] Add Application Tag column to RM All Applications table
[ https://issues.apache.org/jira/browse/YARN-9889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952863#comment-16952863 ] Kinga Marton edited comment on YARN-9889 at 10/16/19 2:08 PM: -- In patch 002 I have addressed the checkstyle issue. The test failures are unrelated. was (Author: kmarton): IN patch 002 I have addressed the checkstyle issue. The test failures are unrelated. > [UI] Add Application Tag column to RM All Applications table > > > Key: YARN-9889 > URL: https://issues.apache.org/jira/browse/YARN-9889 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: AllApplications_with_ApplicationTag.png, > YARN-9889.001.patch, YARN-9889.002.patch > > > Right now AFAIK there is no possibility to filter the applications based on > the application tag in the UI. Adding this new column to the app table will > make this filtering possible as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9889) [UI] Add Application Tag column to RM All Applications table
[ https://issues.apache.org/jira/browse/YARN-9889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952863#comment-16952863 ] Kinga Marton commented on YARN-9889: IN patch 002 I have addressed the checkstyle issue. The test failures are unrelated. > [UI] Add Application Tag column to RM All Applications table > > > Key: YARN-9889 > URL: https://issues.apache.org/jira/browse/YARN-9889 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: AllApplications_with_ApplicationTag.png, > YARN-9889.001.patch, YARN-9889.002.patch > > > Right now AFAIK there is no possibility to filter the applications based on > the application tag in the UI. Adding this new column to the app table will > make this filtering possible as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9889) [UI] Add Application Tag column to RM All Applications table
[ https://issues.apache.org/jira/browse/YARN-9889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-9889: --- Attachment: YARN-9889.002.patch > [UI] Add Application Tag column to RM All Applications table > > > Key: YARN-9889 > URL: https://issues.apache.org/jira/browse/YARN-9889 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: AllApplications_with_ApplicationTag.png, > YARN-9889.001.patch, YARN-9889.002.patch > > > Right now AFAIK there is no possibility to filter the applications based on > the application tag in the UI. Adding this new column to the app table will > make this filtering possible as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9890) [UI2] Add Application tag to the app table and app detail page.
[ https://issues.apache.org/jira/browse/YARN-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952756#comment-16952756 ] Kinga Marton commented on YARN-9890: In the uploaded patch I have extended the applications table with the application tag column. I have also checked the application details page, and I am not sure that we need there this information as well. [~wangda], [~sunilg] what do you think? > [UI2] Add Application tag to the app table and app detail page. > --- > > Key: YARN-9890 > URL: https://issues.apache.org/jira/browse/YARN-9890 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: UI2_ApplicationTag.png, YARN-9890.001.patch > > > Right now AFAIK there is no possibility to filter the applications based on > the application tag in the UI. Adding this new column to the app table will > make this filtering possible as well. > From the UI2 this information is missing from the application detail page as > well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9890) [UI2] Add Application tag to the app table and app detail page.
[ https://issues.apache.org/jira/browse/YARN-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-9890: --- Attachment: YARN-9890.001.patch > [UI2] Add Application tag to the app table and app detail page. > --- > > Key: YARN-9890 > URL: https://issues.apache.org/jira/browse/YARN-9890 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: UI2_ApplicationTag.png, YARN-9890.001.patch > > > Right now AFAIK there is no possibility to filter the applications based on > the application tag in the UI. Adding this new column to the app table will > make this filtering possible as well. > From the UI2 this information is missing from the application detail page as > well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9890) [UI2] Add Application tag to the app table and app detail page.
[ https://issues.apache.org/jira/browse/YARN-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-9890: --- Attachment: UI2_ApplicationTag.png > [UI2] Add Application tag to the app table and app detail page. > --- > > Key: YARN-9890 > URL: https://issues.apache.org/jira/browse/YARN-9890 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: UI2_ApplicationTag.png > > > Right now AFAIK there is no possibility to filter the applications based on > the application tag in the UI. Adding this new column to the app table will > make this filtering possible as well. > From the UI2 this information is missing from the application detail page as > well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9886) Queue mapping based on userid passed through application tag
[ https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952736#comment-16952736 ] Kinga Marton commented on YARN-9886: Sure, [~leftnoteasy]. I have converted them to separate issue. > Queue mapping based on userid passed through application tag > > > Key: YARN-9886 > URL: https://issues.apache.org/jira/browse/YARN-9886 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-9886-WIP.patch > > > There are situations when the real submitting user differs from the user what > arrives to YARN. For example in case of a Hive application when Hive > impersonation is turned off, the hive queries will run as Hive user and the > mapping is done based on this username. Unfortunately in this case YARN > doesn't have any information about the real user and there are cases when the > customer may want to map this applications to the real submitting user's > queue instead of the Hive one. > For this cases if they would pass the username in the application tag we may > read it and use that one during the queue mapping, if that user has rights to > run on the real user's queue. > [~sunilg] please correct me if I missed something. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9890) [UI2] Add Application tag to the app table and app detail page.
[ https://issues.apache.org/jira/browse/YARN-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-9890: --- Parent: (was: YARN-9886) Issue Type: Improvement (was: Sub-task) > [UI2] Add Application tag to the app table and app detail page. > --- > > Key: YARN-9890 > URL: https://issues.apache.org/jira/browse/YARN-9890 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > > Right now AFAIK there is no possibility to filter the applications based on > the application tag in the UI. Adding this new column to the app table will > make this filtering possible as well. > From the UI2 this information is missing from the application detail page as > well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9889) [UI] Add Application Tag column to RM All Applications table
[ https://issues.apache.org/jira/browse/YARN-9889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-9889: --- Parent: (was: YARN-9886) Issue Type: Improvement (was: Sub-task) > [UI] Add Application Tag column to RM All Applications table > > > Key: YARN-9889 > URL: https://issues.apache.org/jira/browse/YARN-9889 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: AllApplications_with_ApplicationTag.png, > YARN-9889.001.patch > > > Right now AFAIK there is no possibility to filter the applications based on > the application tag in the UI. Adding this new column to the app table will > make this filtering possible as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9889) [UI] Add Application Tag column to RM All Applications table
[ https://issues.apache.org/jira/browse/YARN-9889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-9889: --- Attachment: YARN-9889.001.patch > [UI] Add Application Tag column to RM All Applications table > > > Key: YARN-9889 > URL: https://issues.apache.org/jira/browse/YARN-9889 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: AllApplications_with_ApplicationTag.png, > YARN-9889.001.patch > > > Right now AFAIK there is no possibility to filter the applications based on > the application tag in the UI. Adding this new column to the app table will > make this filtering possible as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9889) [UI] Add Application Tag column to RM All Applications table
[ https://issues.apache.org/jira/browse/YARN-9889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-9889: --- Attachment: AllApplications_with_ApplicationTag.png > [UI] Add Application Tag column to RM All Applications table > > > Key: YARN-9889 > URL: https://issues.apache.org/jira/browse/YARN-9889 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: AllApplications_with_ApplicationTag.png > > > Right now AFAIK there is no possibility to filter the applications based on > the application tag in the UI. Adding this new column to the app table will > make this filtering possible as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9886) Queue mapping based on userid passed through application tag
[ https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951012#comment-16951012 ] Kinga Marton commented on YARN-9886: I have uploaded a WIP patch where I have checked if in the application tag is some userId passed and used that one for placing the application. To be honest I don't really like this approach of using the application tags. What if some user has some application tags that will match to our pattern, but he/she does not want to use that userid while mapping the application to the queue. I think that a cleaner solution would be to introduce a new property for this purpose. This way we can be sure that we are not breaking incompatibility and we can avoid any kind of side effects. [~snemeth], [~sunilg] what do you think about introducing a new property instead of using the application tags? > Queue mapping based on userid passed through application tag > > > Key: YARN-9886 > URL: https://issues.apache.org/jira/browse/YARN-9886 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-9886-WIP.patch > > > There are situations when the real submitting user differs from the user what > arrives to YARN. For example in case of a Hive application when Hive > impersonation is turned off, the hive queries will run as Hive user and the > mapping is done based on this username. Unfortunately in this case YARN > doesn't have any information about the real user and there are cases when the > customer may want to map this applications to the real submitting user's > queue instead of the Hive one. > For this cases if they would pass the username in the application tag we may > read it and use that one during the queue mapping, if that user has rights to > run on the real user's queue. > [~sunilg] please correct me if I missed something. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9886) Queue mapping based on userid passed through application tag
[ https://issues.apache.org/jira/browse/YARN-9886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-9886: --- Attachment: YARN-9886-WIP.patch > Queue mapping based on userid passed through application tag > > > Key: YARN-9886 > URL: https://issues.apache.org/jira/browse/YARN-9886 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-9886-WIP.patch > > > There are situations when the real submitting user differs from the user what > arrives to YARN. For example in case of a Hive application when Hive > impersonation is turned off, the hive queries will run as Hive user and the > mapping is done based on this username. Unfortunately in this case YARN > doesn't have any information about the real user and there are cases when the > customer may want to map this applications to the real submitting user's > queue instead of the Hive one. > For this cases if they would pass the username in the application tag we may > read it and use that one during the queue mapping, if that user has rights to > run on the real user's queue. > [~sunilg] please correct me if I missed something. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9890) [UI2] Add Application tag to the app table and app detail page.
Kinga Marton created YARN-9890: -- Summary: [UI2] Add Application tag to the app table and app detail page. Key: YARN-9890 URL: https://issues.apache.org/jira/browse/YARN-9890 Project: Hadoop YARN Issue Type: Sub-task Reporter: Kinga Marton Assignee: Kinga Marton Right now AFAIK there is no possibility to filter the applications based on the application tag in the UI. Adding this new column to the app table will make this filtering possible as well. >From the UI2 this information is missing from the application detail page as >well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9889) [UI] Add Application Tag column to RM All Applications table
Kinga Marton created YARN-9889: -- Summary: [UI] Add Application Tag column to RM All Applications table Key: YARN-9889 URL: https://issues.apache.org/jira/browse/YARN-9889 Project: Hadoop YARN Issue Type: Sub-task Reporter: Kinga Marton Assignee: Kinga Marton Right now AFAIK there is no possibility to filter the applications based on the application tag in the UI. Adding this new column to the app table will make this filtering possible as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9886) Queue mapping based on userid passed through application tag
Kinga Marton created YARN-9886: -- Summary: Queue mapping based on userid passed through application tag Key: YARN-9886 URL: https://issues.apache.org/jira/browse/YARN-9886 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Reporter: Kinga Marton Assignee: Kinga Marton There are situations when the real submitting user differs from the user what arrives to YARN. For example in case of a Hive application when Hive impersonation is turned off, the hive queries will run as Hive user and the mapping is done based on this username. Unfortunately in this case YARN doesn't have any information about the real user and there are cases when the customer may want to map this applications to the real submitting user's queue instead of the Hive one. For this cases if they would pass the username in the application tag we may read it and use that one during the queue mapping, if that user has rights to run on the real user's queue. [~sunilg] please correct me if I missed something. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9883) Reshape SchedulerHealth class
[
https://issues.apache.org/jira/browse/YARN-9883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kinga Marton reassigned YARN-9883:
--
Assignee: Kinga Marton
> Reshape SchedulerHealth class
> -
>
> Key: YARN-9883
> URL: https://issues.apache.org/jira/browse/YARN-9883
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Kinga Marton
>Priority: Minor
>
> The {{SchedulerHealth}} class has some flaws, for example:
> - It has no javadoc at all
> - All its objects are package-private: they should be private
> - The internal maps should be (Concurrent) EnumMaps instead of HashMaps: they
> are more efficient in storing Enums
> - schedulerHealthDetails only stores the last operation, its name should
> reflect that (just like lastSchedulerRunDetails)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9743) [JDK11] TestTimelineWebServices.testContextFactory fails
[
https://issues.apache.org/jira/browse/YARN-9743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kinga Marton reassigned YARN-9743:
--
Assignee: Kinga Marton
> [JDK11] TestTimelineWebServices.testContextFactory fails
>
>
> Key: YARN-9743
> URL: https://issues.apache.org/jira/browse/YARN-9743
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineservice
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Kinga Marton
>Priority: Major
>
> Tested on OpenJDK 11.0.2 on a Mac.
> Stack trace:
> {noformat}
> [ERROR] Tests run: 29, Failures: 0, Errors: 3, Skipped: 0, Time elapsed:
> 36.016 s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
> [ERROR]
> testContextFactory(org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices)
> Time elapsed: 1.031 s <<< ERROR!
> java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:315)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.newContext(ContextFactory.java:85)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.createContext(ContextFactory.java:112)
> at
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices.testContextFactory(TestTimelineWebServices.java:1039)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9784) org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue is flaky
[
https://issues.apache.org/jira/browse/YARN-9784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922205#comment-16922205
]
Julia Kinga Marton commented on YARN-9784:
--
Thank you [~adam.antal] and [~sunilg] for the review!
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
> is flaky
> ---
>
> Key: YARN-9784
> URL: https://issues.apache.org/jira/browse/YARN-9784
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.3.0
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: YARN-9784.001.patch
>
>
> There are some test cases in TestLeafQueue which are failing intermittently.
> From 100 runs, there were 16 failures.
> Some failure examples are the following ones:
> {code:java}
> 2019-08-26 13:18:13 [ERROR] Errors:
> 2019-08-26 13:18:13 [ERROR] TestLeafQueue.setUp:144->setUpInternal:221
> WrongTypeOfReturnValue
> 2019-08-26 13:18:13 YarnConfigu...
> 2019-08-26 13:18:13 [ERROR] TestLeafQueue.setUp:144->setUpInternal:221
> WrongTypeOfReturnValue
> 2019-08-26 13:18:13 YarnConfigu...
> 2019-08-26 13:18:13 [INFO]
> 2019-08-26 13:18:13 [ERROR] Tests run: 36, Failures: 0, Errors: 2, Skipped: 0
> {code}
> {code:java}
> 2019-08-26 13:18:09 [ERROR] Failures:
> 2019-08-26 13:18:09 [ERROR] TestLeafQueue.testHeadroomWithMaxCap:1373
> expected:<2048> but was:<0>
> 2019-08-26 13:18:09 [INFO]
> 2019-08-26 13:18:09 [ERROR] Tests run: 36, Failures: 1, Errors: 0, Skipped: 0
> {code}
> {code:java}
> 2019-08-26 13:18:18 [ERROR] Errors:
> 2019-08-26 13:18:18 [ERROR] TestLeafQueue.setUp:144->setUpInternal:221
> WrongTypeOfReturnValue
> 2019-08-26 13:18:18 YarnConfigu...
> 2019-08-26 13:18:18 [ERROR] TestLeafQueue.testHeadroomWithMaxCap:1307 ?
> ClassCast org.apache.hadoop.yarn.c...
> 2019-08-26 13:18:18 [INFO]
> 2019-08-26 13:18:18 [ERROR] Tests run: 36, Failures: 0, Errors: 2, Skipped: 0
> {code}
> {code:java}
> 2019-08-26 13:18:10 [ERROR] Failures:
> 2019-08-26 13:18:10 [ERROR] TestLeafQueue.testDRFUserLimits:847 Verify
> user_0 got resources
> 2019-08-26 13:18:10 [INFO]
> 2019-08-26 13:18:10 [ERROR] Tests run: 36, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9784) org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue is flaky
[
https://issues.apache.org/jira/browse/YARN-9784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920784#comment-16920784
]
Julia Kinga Marton edited comment on YARN-9784 at 9/2/19 11:28 AM:
---
Test results before and after patch (I have run TestLeafQueue 500 times):
Before patch: 500/500 tests complete (265 failed)
After patch: 500/500 tests complete (0 failed)
was (Author: kmarton):
Test results before and after patch:
Before patch: 500/500 tests complete (265 failed)
After patch: 500/500 tests complete (0 failed)
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
> is flaky
> ---
>
> Key: YARN-9784
> URL: https://issues.apache.org/jira/browse/YARN-9784
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.3.0
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: YARN-9784.001.patch
>
>
> There are some test cases in TestLeafQueue which are failing intermittently.
> From 100 runs, there were 16 failures.
> Some failure examples are the following ones:
> {code:java}
> 2019-08-26 13:18:13 [ERROR] Errors:
> 2019-08-26 13:18:13 [ERROR] TestLeafQueue.setUp:144->setUpInternal:221
> WrongTypeOfReturnValue
> 2019-08-26 13:18:13 YarnConfigu...
> 2019-08-26 13:18:13 [ERROR] TestLeafQueue.setUp:144->setUpInternal:221
> WrongTypeOfReturnValue
> 2019-08-26 13:18:13 YarnConfigu...
> 2019-08-26 13:18:13 [INFO]
> 2019-08-26 13:18:13 [ERROR] Tests run: 36, Failures: 0, Errors: 2, Skipped: 0
> {code}
> {code:java}
> 2019-08-26 13:18:09 [ERROR] Failures:
> 2019-08-26 13:18:09 [ERROR] TestLeafQueue.testHeadroomWithMaxCap:1373
> expected:<2048> but was:<0>
> 2019-08-26 13:18:09 [INFO]
> 2019-08-26 13:18:09 [ERROR] Tests run: 36, Failures: 1, Errors: 0, Skipped: 0
> {code}
> {code:java}
> 2019-08-26 13:18:18 [ERROR] Errors:
> 2019-08-26 13:18:18 [ERROR] TestLeafQueue.setUp:144->setUpInternal:221
> WrongTypeOfReturnValue
> 2019-08-26 13:18:18 YarnConfigu...
> 2019-08-26 13:18:18 [ERROR] TestLeafQueue.testHeadroomWithMaxCap:1307 ?
> ClassCast org.apache.hadoop.yarn.c...
> 2019-08-26 13:18:18 [INFO]
> 2019-08-26 13:18:18 [ERROR] Tests run: 36, Failures: 0, Errors: 2, Skipped: 0
> {code}
> {code:java}
> 2019-08-26 13:18:10 [ERROR] Failures:
> 2019-08-26 13:18:10 [ERROR] TestLeafQueue.testDRFUserLimits:847 Verify
> user_0 got resources
> 2019-08-26 13:18:10 [INFO]
> 2019-08-26 13:18:10 [ERROR] Tests run: 36, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9784) org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue is flaky
[
https://issues.apache.org/jira/browse/YARN-9784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julia Kinga Marton updated YARN-9784:
-
Summary:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
is flaky (was: TestLeafQueue is flaky)
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
> is flaky
> ---
>
> Key: YARN-9784
> URL: https://issues.apache.org/jira/browse/YARN-9784
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.3.0
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: YARN-9784.001.patch
>
>
> There are some test cases in TestLeafQueue which are failing intermittently.
> From 100 runs, there were 16 failures.
> Some failure examples are the following ones:
> {code:java}
> 2019-08-26 13:18:13 [ERROR] Errors:
> 2019-08-26 13:18:13 [ERROR] TestLeafQueue.setUp:144->setUpInternal:221
> WrongTypeOfReturnValue
> 2019-08-26 13:18:13 YarnConfigu...
> 2019-08-26 13:18:13 [ERROR] TestLeafQueue.setUp:144->setUpInternal:221
> WrongTypeOfReturnValue
> 2019-08-26 13:18:13 YarnConfigu...
> 2019-08-26 13:18:13 [INFO]
> 2019-08-26 13:18:13 [ERROR] Tests run: 36, Failures: 0, Errors: 2, Skipped: 0
> {code}
> {code:java}
> 2019-08-26 13:18:09 [ERROR] Failures:
> 2019-08-26 13:18:09 [ERROR] TestLeafQueue.testHeadroomWithMaxCap:1373
> expected:<2048> but was:<0>
> 2019-08-26 13:18:09 [INFO]
> 2019-08-26 13:18:09 [ERROR] Tests run: 36, Failures: 1, Errors: 0, Skipped: 0
> {code}
> {code:java}
> 2019-08-26 13:18:18 [ERROR] Errors:
> 2019-08-26 13:18:18 [ERROR] TestLeafQueue.setUp:144->setUpInternal:221
> WrongTypeOfReturnValue
> 2019-08-26 13:18:18 YarnConfigu...
> 2019-08-26 13:18:18 [ERROR] TestLeafQueue.testHeadroomWithMaxCap:1307 ?
> ClassCast org.apache.hadoop.yarn.c...
> 2019-08-26 13:18:18 [INFO]
> 2019-08-26 13:18:18 [ERROR] Tests run: 36, Failures: 0, Errors: 2, Skipped: 0
> {code}
> {code:java}
> 2019-08-26 13:18:10 [ERROR] Failures:
> 2019-08-26 13:18:10 [ERROR] TestLeafQueue.testDRFUserLimits:847 Verify
> user_0 got resources
> 2019-08-26 13:18:10 [INFO]
> 2019-08-26 13:18:10 [ERROR] Tests run: 36, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
