[jira] [Created] (YARN-9039) App ACLs are not validated when serving logs from Logs CLI/Yarn UI2
Suma Shivaprasad created YARN-9039: -- Summary: App ACLs are not validated when serving logs from Logs CLI/Yarn UI2 Key: YARN-9039 URL: https://issues.apache.org/jira/browse/YARN-9039 Project: Hadoop YARN Issue Type: Bug Components: log-aggregation Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad Attachments: YARN-9039.1.patch App Acls are not being validated when serving logs through YARN CLI. This also applies while serving logs through YARN UIV2 through ATSV2 Log Webservice -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9030) Log aggregation changes to handle filesystems which do not support permissions
Suma Shivaprasad created YARN-9030:
--
Summary: Log aggregation changes to handle filesystems which do
not support permissions
Key: YARN-9030
URL: https://issues.apache.org/jira/browse/YARN-9030
Project: Hadoop YARN
Issue Type: Bug
Reporter: Suma Shivaprasad
Assignee: Suma Shivaprasad
Some cloud storages like ABFS do not support permissions in which case they
throw an UnsupportedOperationException. Log aggregation should hanlde these
case and not set permissions for log aggregation base dir/ sub dirs
{noformat}
2018-11-12 15:37:28,726 WARN logaggregation.LogAggregationService
(LogAggregationService.java:initApp(209)) - Application failed to init
aggregation
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to check
permissions for dir [abfs://testc...@test.blob.core.windows.net/app-logs]
at
org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController.verifyAndCreateRemoteLogDir(LogAggregationFileController.java:277)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initAppAggregator(LogAggregationService.java:238)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:204)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:347)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:69)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4249) Many options in "yarn application" command is not documented
[
https://issues.apache.org/jira/browse/YARN-4249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suma Shivaprasad resolved YARN-4249.
Resolution: Not A Problem
> Many options in "yarn application" command is not documented
>
>
> Key: YARN-4249
> URL: https://issues.apache.org/jira/browse/YARN-4249
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: nijel
>Priority: Major
>
> in document only few options are specified.
> {code}
> Usage: `yarn application [options] `
> | COMMAND\_OPTIONS | Description |
> |: |: |
> | -appStates \ | Works with -list to filter applications based on
> input comma-separated list of application states. The valid application state
> can be one of the following: ALL, NEW, NEW\_SAVING, SUBMITTED, ACCEPTED,
> RUNNING, FINISHED, FAILED, KILLED |
> | -appTypes \ | Works with -list to filter applications based on
> input comma-separated list of application types. |
> | -list | Lists applications from the RM. Supports optional use of -appTypes
> to filter applications based on application type, and -appStates to filter
> applications based on application state. |
> | -kill \ | Kills the application. |
> | -status \ | Prints the status of the application. |
> {code}
> some options are missing like
> -appId Specify Application Id to be operated
> -help Displays help for all commands.
> -movetoqueueMoves the application to a different queue.
> -queue Works with the movetoqueue command to specify
> which queue to move an application to.
> -updatePriority update priority of an
> application.ApplicationId can be passed using 'appId' option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8923) Cleanup references to ENV file type in code
Suma Shivaprasad created YARN-8923: -- Summary: Cleanup references to ENV file type in code Key: YARN-8923 URL: https://issues.apache.org/jira/browse/YARN-8923 Project: Hadoop YARN Issue Type: Improvement Components: yarn-native-services Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad YARN Service code still has references to ENV file type which should be removed. And a few other unused methods in configuration file handling which could be removed -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8920) LogAggregation should be configurable to allow writing to underlying storage as appOwner or yarn user
Suma Shivaprasad created YARN-8920: -- Summary: LogAggregation should be configurable to allow writing to underlying storage as appOwner or yarn user Key: YARN-8920 URL: https://issues.apache.org/jira/browse/YARN-8920 Project: Hadoop YARN Issue Type: Improvement Components: log-aggregation, yarn Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8711) YARN UI2 : Display component state in Component list and details page for a Service
Suma Shivaprasad created YARN-8711: -- Summary: YARN UI2 : Display component state in Component list and details page for a Service Key: YARN-8711 URL: https://issues.apache.org/jira/browse/YARN-8711 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Reporter: Suma Shivaprasad Assignee: Akhil PB YARN-8488 adds component states and service state = SUCCEEDED. Users could then track overall component status on the UI for terminating jobs. cc [~sunil.gov...@gmail.com] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8710) Service AM should set a finite limit on NM container max retries
Suma Shivaprasad created YARN-8710: -- Summary: Service AM should set a finite limit on NM container max retries Key: YARN-8710 URL: https://issues.apache.org/jira/browse/YARN-8710 Project: Hadoop YARN Issue Type: Bug Components: yarn-native-services Environment: Container retries are currently set to a default of -1 in AbstractProviderService.buildContainerRetry. If this is not overridden via service spec with a finite value for yarn.service.container-failure.retry.max , this causes infinite NM reties for the container for ALWAYS/ON_FAILURE restart policy . Ideally it should try a finite number of time on the same NM and subsequently Service AM can retry on another node. We can set this to default value of 3. Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8675) Setting hostname of docker container breaks with "host" networking mode for Apps which do not run as a YARN service
Suma Shivaprasad created YARN-8675: -- Summary: Setting hostname of docker container breaks with "host" networking mode for Apps which do not run as a YARN service Key: YARN-8675 URL: https://issues.apache.org/jira/browse/YARN-8675 Project: Hadoop YARN Issue Type: Bug Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad Applications like the Spark AM currently do not run as a YARN service and setting hostname breaks driver/executor communication if docker version >=1.13.1 , especially with wire-encryption turned on. The proposal is to not set the hostname when "host" networking mode is enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8588) Logging improvements for better debuggability
Suma Shivaprasad created YARN-8588: -- Summary: Logging improvements for better debuggability Key: YARN-8588 URL: https://issues.apache.org/jira/browse/YARN-8588 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad Capacity allocations decided in GuaranteedCapacityOvertimePolicy are available via AutoCreatedLeafQueueConfig. However this class lacks a toString and some other DEBUG level logs are needed for better debuggability -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8572) YarnClient getContainers API should support filtering by container status
Suma Shivaprasad created YARN-8572: -- Summary: YarnClient getContainers API should support filtering by container status Key: YARN-8572 URL: https://issues.apache.org/jira/browse/YARN-8572 Project: Hadoop YARN Issue Type: Improvement Components: yarn Reporter: Suma Shivaprasad YarnClient.getContainers should support filtering containers by their status - RUNNING, COMPLETED etc . This may require corresponding changes in ATS to filter by container status for a given application attemopt -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8519) Yarn UI2 : Changes to depict Auto Created leaf Queues/Managed Queues differently from other queues
Suma Shivaprasad created YARN-8519: -- Summary: Yarn UI2 : Changes to depict Auto Created leaf Queues/Managed Queues differently from other queues Key: YARN-8519 URL: https://issues.apache.org/jira/browse/YARN-8519 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad YARN-7420 covers changes to depict auto created leaf queues in a separate color notation but this was done in the old Yarn UI and similiar chnages need to be incorporated in the new YARN UI to depict Managed Parent queues/Auto-Created leaf queues separately -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8419) In "New Service" section of new YARN UI, user cannot submit service as Submit button is always disabled.
Suma Shivaprasad created YARN-8419: -- Summary: In "New Service" section of new YARN UI, user cannot submit service as Submit button is always disabled. Key: YARN-8419 URL: https://issues.apache.org/jira/browse/YARN-8419 Project: Hadoop YARN Issue Type: Bug Reporter: Suma Shivaprasad This is because, user.name check is still mandatory for non-secure cluster. But in secure cluster, user.name is not exposed to UI and hence was showing this error in secure cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8415) TimelineWebServices should throw a ForBiddenException ( 403
Suma Shivaprasad created YARN-8415: -- Summary: TimelineWebServices should throw a ForBiddenException ( 403 Key: YARN-8415 URL: https://issues.apache.org/jira/browse/YARN-8415 Project: Hadoop YARN Issue Type: Bug Reporter: Suma Shivaprasad -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8339) Service AM should localize static/archive resource types to container working directory instead of 'resources' diurectory
Suma Shivaprasad created YARN-8339: -- Summary: Service AM should localize static/archive resource types to container working directory instead of 'resources' diurectory Key: YARN-8339 URL: https://issues.apache.org/jira/browse/YARN-8339 Project: Hadoop YARN Issue Type: Bug Components: yarn-native-services Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad This is to address one of the review comments posted by [~wangda] in YARN-8079 at https://issues.apache.org/jira/browse/YARN-8079?focusedCommentId=16482065=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16482065 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8300) Fix NPE in DefaultUpgradeComponentsFinder
Suma Shivaprasad created YARN-8300:
--
Summary: Fix NPE in DefaultUpgradeComponentsFinder
Key: YARN-8300
URL: https://issues.apache.org/jira/browse/YARN-8300
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Reporter: Suma Shivaprasad
Assignee: Suma Shivaprasad
In current upgrades for Yarn native services, we do not support
addition/deletion of compoents during upgrade. On trying to upgrade with the
same number of components in target spec as the current service spec but with
the one of the components having a new target spec and name, see the following
NPE in service AM logs
{noformat}
2018-05-15 00:10:41,489 [IPC Server handler 0 on 37488] ERROR
service.ClientAMService - Error while trying to upgrade service {}
java.lang.NullPointerException
at
org.apache.hadoop.yarn.service.UpgradeComponentsFinder$DefaultUpgradeComponentsFinder.lambda$findTargetComponentSpecs$0(UpgradeComponentsFinder.java:103)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at
org.apache.hadoop.yarn.service.UpgradeComponentsFinder$DefaultUpgradeComponentsFinder.findTargetComponentSpecs(UpgradeComponentsFinder.java:100)
at
org.apache.hadoop.yarn.service.ServiceManager.processUpgradeRequest(ServiceManager.java:259)
at
org.apache.hadoop.yarn.service.ClientAMService.upgrade(ClientAMService.java:163)
at
org.apache.hadoop.yarn.service.impl.pb.service.ClientAMProtocolPBServiceImpl.upgradeService(ClientAMProtocolPBServiceImpl.java:81)
at
org.apache.hadoop.yarn.proto.ClientAMProtocol$ClientAMProtocolService$2.callBlockingMethod(ClientAMProtocol.java:5972)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8296) Update YarnServiceApi documentation and yarn service UI code to remove references to unique_component_support
Suma Shivaprasad created YARN-8296: -- Summary: Update YarnServiceApi documentation and yarn service UI code to remove references to unique_component_support Key: YARN-8296 URL: https://issues.apache.org/jira/browse/YARN-8296 Project: Hadoop YARN Issue Type: Bug Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad yarn-servicedef.js/service-component-table.hbs and YarnServiceAPI.md has references to unique_component_support which is unused in Yarn service implementation code. We should remove this from docs/UI code if this is no longer used. cc [~billie.rinaldi] [~gsaha] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8255) Allow option to disable flex for a service component
Suma Shivaprasad created YARN-8255: -- Summary: Allow option to disable flex for a service component Key: YARN-8255 URL: https://issues.apache.org/jira/browse/YARN-8255 Project: Hadoop YARN Issue Type: Bug Components: yarn-native-services Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad YARN-8080 implements restart capabilities for service component instances. YARN service components should add an option to disallow flexing to support workloads which are essentially batch/iterative jobs which terminate with restart_policy=NEVER/ON_FAILURE. This could be disabled by default for components where restart_policy=NEVER/ON_FAILURE and enabled by default when restart_policy=ALWAYS(which is the default restart_policy) unless explicitly set at the service spec. The option could be exposed as part of the component spec as "allow_flexing". cc [~billie.rinaldi] [~gsaha] [~eyang] [~csingh] [~wangda] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8217) RmAuthenticationFilterInitializer /TimelineAuthenticationFilterInitializer should use Configuration.getPropsWithPrefix instead of iterator
Suma Shivaprasad created YARN-8217: -- Summary: RmAuthenticationFilterInitializer /TimelineAuthenticationFilterInitializer should use Configuration.getPropsWithPrefix instead of iterator Key: YARN-8217 URL: https://issues.apache.org/jira/browse/YARN-8217 Project: Hadoop YARN Issue Type: Bug Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad HADOOP-15411 fixed a similar issue for AuthenticationFilterInitializer. This issue can occur in RmAuthenticationFilterInitializer/TimelineAuthenticationFilterInitializer as well -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8210) AMRMClient logging on every heartbeat to track updation of AM RM token causes too many log lines to be generated in AM logs
Suma Shivaprasad created YARN-8210: -- Summary: AMRMClient logging on every heartbeat to track updation of AM RM token causes too many log lines to be generated in AM logs Key: YARN-8210 URL: https://issues.apache.org/jira/browse/YARN-8210 Project: Hadoop YARN Issue Type: Bug Components: yarn Affects Versions: 3.0.0-alpha1, 2.9.0 Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad YARN-4682 added logs to track when AM RM token is updated for debuggability purposes. However this is printed on every heartbeat and could cause the AM logs to be flooded with this whenever RM's Master key is rolled over especially if its a long running AM. Hence proposing to remove this log line -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8177) Fix documentation for node label support
Suma Shivaprasad created YARN-8177: -- Summary: Fix documentation for node label support Key: YARN-8177 URL: https://issues.apache.org/jira/browse/YARN-8177 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad Capacity Scheduler Dynamic Queues feature documentation needs to be fixed for node label support with examples. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8121) Container runtime and runtime context information should be available in UI
Suma Shivaprasad created YARN-8121: -- Summary: Container runtime and runtime context information should be available in UI Key: YARN-8121 URL: https://issues.apache.org/jira/browse/YARN-8121 Project: Hadoop YARN Issue Type: Improvement Reporter: Suma Shivaprasad Currently we do not publish container runtime and other runtime context information(important in case of docker ) - like the the docker image used, docker network the container ran on etc in ATS. This needs to be added in ATS and made available in UI to provide visibility on container runtime chosen and the runtime parameters to end user -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8086) ManagedParentQueue with no leaf queues cause JS error in new UI
Suma Shivaprasad created YARN-8086: -- Summary: ManagedParentQueue with no leaf queues cause JS error in new UI Key: YARN-8086 URL: https://issues.apache.org/jira/browse/YARN-8086 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.1.0 Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad ManagedParentQueues do not have any leaf queues and capacity-queue.js does not do null checks while checking for child queues under Managed parent queues while displaying "Cluster Overview/Queues" page if such queues exist in capacity-scheduler configuration -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7987) Docker container name(--name) needs to be DNS friendly for DNS resolution to work in user defined networks.
[ https://issues.apache.org/jira/browse/YARN-7987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad resolved YARN-7987. Resolution: Won't Fix > Docker container name(--name) needs to be DNS friendly for DNS resolution to > work in user defined networks. > > > Key: YARN-7987 > URL: https://issues.apache.org/jira/browse/YARN-7987 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Major > > User defined networks like overlays support DNS resolution through Docker > Embedded DNS which needs the container name (–name parameter value in docker > run) to be a FQDN for container names to be resolved - Please refer > documentation > [https://docs.docker.com/v17.09/engine/userguide/networking/configure-dns/] > However Yarn sets the container name to the container's id which is not DNS > friendly(eg: container_e26_1519402686002_0035_01_03) and is not a FQDN. > The proposal is to set a FQDN(eg: > ctr-e26-1519402686002-0035-01-03.domain-name) as the docker container's > name for containers to be able to communicate to each other via hostnames in > user defined networks like overlays, bridges etc. The domain name will be > picked up from the YARN DNS registry configuration > (hadoop.registry.dns.domain-name) > > > > > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7994) Add support for network-alias in docker run for user defined networks
Suma Shivaprasad created YARN-7994:
--
Summary: Add support for network-alias in docker run for user
defined networks
Key: YARN-7994
URL: https://issues.apache.org/jira/browse/YARN-7994
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Suma Shivaprasad
Assignee: Suma Shivaprasad
Docker Embedded DNS supports DNS resolution for containers by one or more of
its configured {{--network-alias}} within a user-defined network.
DockerRunCommand should support this option for DNS resolution to work through
docker embedded DNS
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7987) Docker container name(--name) needs to be DNS friendly for DNS resolution to work in user defined networks.
Suma Shivaprasad created YARN-7987: -- Summary: Docker container name(--name) needs to be DNS friendly for DNS resolution to work in user defined networks. Key: YARN-7987 URL: https://issues.apache.org/jira/browse/YARN-7987 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad User defined networks like overlays support DNS resolution through Docker Embedded DNS which needs the container name (–name parameter value in docker run) to be a FQDN for container names to be resolved - Please refer documentation [https://docs.docker.com/v17.09/engine/userguide/networking/configure-dns/] However Yarn sets the container name to the container's id which is not DNS friendly(eg: container_e26_1519402686002_0035_01_03) and is not a FQDN. The proposal is to set a FQDN(eg: ctr-e26-1519402686002-0035-01-03.domain-name) as the docker container's name for containers to be able to communicate to each other via hostnames in user defined networks like overlays, bridges etc. The domain name will be picked up from the YARN DNS registry configuration (hadoop.registry.dns.domain-name) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7935) Expose container's hostname to applications running within the docker container
Suma Shivaprasad created YARN-7935: -- Summary: Expose container's hostname to applications running within the docker container Key: YARN-7935 URL: https://issues.apache.org/jira/browse/YARN-7935 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad Some applications have a need to bind to the container's hostname (like Spark) which is different from the NodeManager's hostname(NM_HOST which is available as an env during container launch) when launched through Docker runtime. The container's hostname can be exposed to applications via an env CONTAINER_HOSTNAME. Another potential candidate is the container's IP but this can be addressed in a separate jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7657) Queue Mapping could provide options to provide 'user' specific auto-created queues under a specified group parent queue
Suma Shivaprasad created YARN-7657: -- Summary: Queue Mapping could provide options to provide 'user' specific auto-created queues under a specified group parent queue Key: YARN-7657 URL: https://issues.apache.org/jira/browse/YARN-7657 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad Current Queue-Mapping only provides %user as an option for 'user' specific queues as u:%user:%user. We can also support %user with group as 'g:makerting-group:marketing.%user' and user specific queues can be automatically created under a group queue in this case. cc [~leftnoteasy] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7643) Handle recovery of applications on auto-created leaf queues
Suma Shivaprasad created YARN-7643: -- Summary: Handle recovery of applications on auto-created leaf queues Key: YARN-7643 URL: https://issues.apache.org/jira/browse/YARN-7643 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad CapacityScheduler application recovery should auto-create leaf queue if it doesnt exist. Also RMAppManager needs to set the queue-mapping placement context so that scheduler has necessary placement context to recreate the queue -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7639) Queue Management scheduling edit policy class needs to be configured dynamically
Suma Shivaprasad created YARN-7639: -- Summary: Queue Management scheduling edit policy class needs to be configured dynamically Key: YARN-7639 URL: https://issues.apache.org/jira/browse/YARN-7639 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad This needs to be configured dynamically for yarn.resourcemanager.monitor.capacity.queue-management.monitoring-interval whenever auto leaf queue creation is enabled for a parent queue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7638) Add unit tests for Preemption and Recovery
Suma Shivaprasad created YARN-7638: -- Summary: Add unit tests for Preemption and Recovery Key: YARN-7638 URL: https://issues.apache.org/jira/browse/YARN-7638 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad Add unit tests to test inter leaf-queue pre-emption based on utilization and work preserving start/recovery. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7634) Queue ACL validations should validate parent queue ACLs before auto-creating leaf queues
Suma Shivaprasad created YARN-7634: -- Summary: Queue ACL validations should validate parent queue ACLs before auto-creating leaf queues Key: YARN-7634 URL: https://issues.apache.org/jira/browse/YARN-7634 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad RMAppManager currently validates only leaf queue ACLs and if leaf queue doesnt exist which is the case in auto-created leaf queues, queue mapping may return a parent queue. However Parent queue ACLs are not validated. This needs to be validated before auto-creating leaf queues for the mapped parent queue -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7633) Add documentation for auto queue creation feature and related configurations
Suma Shivaprasad created YARN-7633: -- Summary: Add documentation for auto queue creation feature and related configurations Key: YARN-7633 URL: https://issues.apache.org/jira/browse/YARN-7633 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7632) Effective min and max resource need to be set for auto created leaf queues upon creation and capacity management
Suma Shivaprasad created YARN-7632: -- Summary: Effective min and max resource need to be set for auto created leaf queues upon creation and capacity management Key: YARN-7632 URL: https://issues.apache.org/jira/browse/YARN-7632 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad YARN-5881 introduced the notion of configuring queues with Absolute resource specifications instead of percentage. As part of that , each leaf queue has an effective min/max capacity that needs to be set when queue is created and whenever queue capacity is changed -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7586) Application Placement should be done before ACL checks in ResourceManager
Suma Shivaprasad created YARN-7586: -- Summary: Application Placement should be done before ACL checks in ResourceManager Key: YARN-7586 URL: https://issues.apache.org/jira/browse/YARN-7586 Project: Hadoop YARN Issue Type: Bug Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad YARN-7473 moved Application placement from RMAppManager to RMAppImpl which causes issues since ApplicationSubmissionContext still has the original queue specified by the user and not the mapped queue . This causes issues while doing ACL checks in RMAppManager -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7574) Add support for Node Labels on Auto Created Leaf Queue Template
Suma Shivaprasad created YARN-7574: -- Summary: Add support for Node Labels on Auto Created Leaf Queue Template Key: YARN-7574 URL: https://issues.apache.org/jira/browse/YARN-7574 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad YARN-7473 adds support for auto created leaf queues to inherit node labels capacities from parent queues. Howebver there is no support for leaf queue template to allow different configured capacities for different node labels. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7520) Queue Ordering policy changes for ordering auto created leaf queues within Managed parent Queues
Suma Shivaprasad created YARN-7520: -- Summary: Queue Ordering policy changes for ordering auto created leaf queues within Managed parent Queues Key: YARN-7520 URL: https://issues.apache.org/jira/browse/YARN-7520 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7473) Implement Framework and policy for capacity management of auto created queues
Suma Shivaprasad created YARN-7473: -- Summary: Implement Framework and policy for capacity management of auto created queues Key: YARN-7473 URL: https://issues.apache.org/jira/browse/YARN-7473 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad This jira mainly addresses the following 1.Support adding pluggable policies on parent queue for dynamically managing capacity/state for leaf queues. 2. Implement a default policy that manages capacity based on pending applications and either grants guaranteed or zero capacity to queues based on parent's available guaranteed capacity. 3. Integrate with SchedulingEditPolicy framework to trigger this periodically and signal scheduler to take necessary actions for capacity/queue management. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7420) YARN UI changes to depict auto created queues
Suma Shivaprasad created YARN-7420: -- Summary: YARN UI changes to depict auto created queues Key: YARN-7420 URL: https://issues.apache.org/jira/browse/YARN-7420 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad Auto created queues will be depicted in a different color to indicate they have been auto created and for easier distinction from manually pre-configured queues. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7419) Implement Auto Queue Creation with modifications to queue mapping flow
Suma Shivaprasad created YARN-7419: -- Summary: Implement Auto Queue Creation with modifications to queue mapping flow Key: YARN-7419 URL: https://issues.apache.org/jira/browse/YARN-7419 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad This involves changes to queue mapping flow to pass along context information for auto queue creation. Auto creation of queues will be part of Capacity Scheduler flow while attempting to resolve queues during application submission. The leaf queues which do not exist are auto created under parent queues which have been explicitly enabled for auto queue creation . In order to determine which parent queue to create the leaf queues under - parent queues need to be specified in queue mapping configuration -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7394) Merge code paths for Reservation/Plan queues and Auto Created queues
Suma Shivaprasad created YARN-7394: -- Summary: Merge code paths for Reservation/Plan queues and Auto Created queues Key: YARN-7394 URL: https://issues.apache.org/jira/browse/YARN-7394 Project: Hadoop YARN Issue Type: Sub-task Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad The initialization/reinitialization logic for ReservationQueue and AutoCreated Leaf queues are similar. The proposal is to rename ReservationQueue to a more generic name AutoCreatedLeafQueue which are either managed by PlanQueue(already exists) or AutoCreatedEnabledParentQueue (new class). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7103) Add logs for errors while signalling containers for better debuggability
Suma Shivaprasad created YARN-7103: -- Summary: Add logs for errors while signalling containers for better debuggability Key: YARN-7103 URL: https://issues.apache.org/jira/browse/YARN-7103 Project: Hadoop YARN Issue Type: Bug Reporter: Suma Shivaprasad -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-3346) Deadlock in Capacity Scheduler
Suma Shivaprasad created YARN-3346:
--
Summary: Deadlock in Capacity Scheduler
Key: YARN-3346
URL: https://issues.apache.org/jira/browse/YARN-3346
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler
Affects Versions: 2.6.0
Reporter: Suma Shivaprasad
{noformat}
Found one Java-level deadlock:
=
2144051991@qtp-383501499-6:
waiting to lock monitor 0x7fa700eec8e8 (object 0x0004589fec18, a
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp),
which is held by ResourceManager Event Processor
ResourceManager Event Processor:
waiting to lock monitor 0x7fa700aadf88 (object 0x000441c05ec8, a
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue),
which is held by IPC Server handler 0 on 54311
IPC Server handler 0 on 54311:
waiting to lock monitor 0x7fa700e20798 (object 0x000441d867f8, a
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue),
which is held by ResourceManager Event Processor
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
