[jira] [Commented] (YARN-7055) YARN Timeline Service v.2: beta 1 / GA
[ https://issues.apache.org/jira/browse/YARN-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918797#comment-16918797 ] Vrushali C commented on YARN-7055: -- Current status: Done: - Migration Path plan for tsv1 to tsv2. - Supporting multiple hbase versions. - Security / Authorization ACLs (at a higher level). [~rohithsharma] will add more details to the epic after consolidating work that has gone into trunk. - Fault tolerance at the process level. [~Prabhu Joseph] [~abmodi] and [~rohithsharma] will consolidate the jiras done towards this effort Coming next: - Security / Authorization ACLs for entities - Fault tolerance at the data level Based on the community discussion call ([~sunil.gov...@gmail.com], [~rohithsharma], [~abmodi], [~prabham] [~Sushil-K-S] [~asadineni] and [~vrushalic]) right now, we are looking at timeline service work that will be a part of the next 3.2.1 release. We will do some documentation updates , some jira cleanup and link all appropriate jiras. We are also going to move some feature jiras to the next epic jira YARN-9802 . > YARN Timeline Service v.2: beta 1 / GA > -- > > Key: YARN-7055 > URL: https://issues.apache.org/jira/browse/YARN-7055 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Priority: Major > Attachments: TSv2 next steps.pdf > > > This is an umbrella JIRA for the beta 1 milestone for YARN Timeline Service > v.2. > YARN-2928 was alpha1, YARN-5355 was alpha2. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7055) YARN Timeline Service v.2: beta 1 / GA
[ https://issues.apache.org/jira/browse/YARN-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918792#comment-16918792 ] Vrushali C commented on YARN-7055: -- Filed YARN-9802 to track features / enhancements to be done after the next release for ATSv2. Also marked this jira as unassigned since it's a community effort. > YARN Timeline Service v.2: beta 1 / GA > -- > > Key: YARN-7055 > URL: https://issues.apache.org/jira/browse/YARN-7055 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Priority: Major > Attachments: TSv2 next steps.pdf > > > This is an umbrella JIRA for the beta 1 milestone for YARN Timeline Service > v.2. > YARN-2928 was alpha1, YARN-5355 was alpha2. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9802) YARN Timeline Service v2 (post GA features)
Vrushali C created YARN-9802: Summary: YARN Timeline Service v2 (post GA features) Key: YARN-9802 URL: https://issues.apache.org/jira/browse/YARN-9802 Project: Hadoop YARN Issue Type: New Feature Reporter: Vrushali C Epic jira to track new feature enhancements or issues after the next release of Timeline Service v2 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7055) YARN Timeline Service v.2: beta 1 / GA
[ https://issues.apache.org/jira/browse/YARN-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-7055: - Summary: YARN Timeline Service v.2: beta 1 / GA (was: YARN Timeline Service v.2: beta 1 ) > YARN Timeline Service v.2: beta 1 / GA > -- > > Key: YARN-7055 > URL: https://issues.apache.org/jira/browse/YARN-7055 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Priority: Major > Attachments: TSv2 next steps.pdf > > > This is an umbrella JIRA for the beta 1 milestone for YARN Timeline Service > v.2. > YARN-2928 was alpha1, YARN-5355 was alpha2. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7055) YARN Timeline Service v.2: beta 1
[ https://issues.apache.org/jira/browse/YARN-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C reassigned YARN-7055: Assignee: (was: Vrushali C) > YARN Timeline Service v.2: beta 1 > -- > > Key: YARN-7055 > URL: https://issues.apache.org/jira/browse/YARN-7055 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Priority: Major > Attachments: TSv2 next steps.pdf > > > This is an umbrella JIRA for the beta 1 milestone for YARN Timeline Service > v.2. > YARN-2928 was alpha1, YARN-5355 was alpha2. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9335) [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls
[ https://issues.apache.org/jira/browse/YARN-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-9335: - Labels: atvs (was: ) > [atsv2] Restrict the number of elements held in timeline collector when > backend is unreachable for async calls > -- > > Key: YARN-9335 > URL: https://issues.apache.org/jira/browse/YARN-9335 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vrushali C >Assignee: Abhishek Modi >Priority: Major > Labels: atvs > Fix For: 3.3.0 > > Attachments: YARN-9335.001.patch, YARN-9335.002.patch, > YARN-9335.003.patch, YARN-9335.004.patch > > > For ATSv2 , if the backend is unreachable, the number/size of data held in > timeline collector's memory increases significantly. This is not good for the > NM memory. > Filing jira to set a limit on how many/much should be retained by the > timeline collector in memory in case the backend is not reachable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9335) [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls
[ https://issues.apache.org/jira/browse/YARN-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811176#comment-16811176 ] Vrushali C commented on YARN-9335: -- Thanks for updating the patch Abhishek!. Committing v4 shortly > [atsv2] Restrict the number of elements held in timeline collector when > backend is unreachable for async calls > -- > > Key: YARN-9335 > URL: https://issues.apache.org/jira/browse/YARN-9335 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vrushali C >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9335.001.patch, YARN-9335.002.patch, > YARN-9335.003.patch, YARN-9335.004.patch > > > For ATSv2 , if the backend is unreachable, the number/size of data held in > timeline collector's memory increases significantly. This is not good for the > NM memory. > Filing jira to set a limit on how many/much should be retained by the > timeline collector in memory in case the backend is not reachable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9382) Publish container killed, paused and resumed events to ATSv2.
[ https://issues.apache.org/jira/browse/YARN-9382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-9382: - Labels: atsv2 (was: ) > Publish container killed, paused and resumed events to ATSv2. > - > > Key: YARN-9382 > URL: https://issues.apache.org/jira/browse/YARN-9382 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Labels: atsv2 > Fix For: 3.3.0 > > Attachments: YARN-9382.001.patch, YARN-9382.002.patch, > YARN-9382.003.patch > > > There are some events missing in container lifecycle. We need to add support > for adding events for when container gets killed, paused and resumed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9382) Publish container killed, paused and resumed events to ATSv2.
[ https://issues.apache.org/jira/browse/YARN-9382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811161#comment-16811161 ] Vrushali C commented on YARN-9382: -- Thanks Abhishek! Committing v03 shortly > Publish container killed, paused and resumed events to ATSv2. > - > > Key: YARN-9382 > URL: https://issues.apache.org/jira/browse/YARN-9382 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9382.001.patch, YARN-9382.002.patch, > YARN-9382.003.patch > > > There are some events missing in container lifecycle. We need to add support > for adding events for when container gets killed, paused and resumed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3488) AM get timeline service info from RM rather than Application specific configuration.
[ https://issues.apache.org/jira/browse/YARN-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809538#comment-16809538 ] Vrushali C commented on YARN-3488: -- Hi Abhishek Yes I will try to get to this soon. > AM get timeline service info from RM rather than Application specific > configuration. > > > Key: YARN-3488 > URL: https://issues.apache.org/jira/browse/YARN-3488 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications >Reporter: Junping Du >Assignee: Abhishek Modi >Priority: Major > Labels: YARN-5355 > Attachments: YARN-3488.001.patch, YARN-3488.002.patch, > YARN-3488.003.patch > > > Since v1 timeline service, we have MR configuration to enable/disable putting > history event to timeline service. For today's v2 timeline service ongoing > effort, currently we have different methods/structures between v1 and v2 for > consuming TimelineClient, so application have to be aware of which version > timeline service get used there. > There are basically two options here: > First option is as current way in DistributedShell or MR to let application > has specific configuration to point out that if enabling ATS and which > version could be, like: MRJobConfig.MAPREDUCE_JOB_EMIT_TIMELINE_DATA, etc. > The other option is to let application to figure out timeline related info > from YARN/RM, it can be done through registerApplicationMaster() in > ApplicationMasterProtocol with return value for service "off", "v1_on", or > "v2_on". > We prefer the latter option because application owner doesn't have to aware > RM/YARN infrastructure details. Please note that we should keep compatible > (consistent behavior with the same setting) with released configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9303) Username splits won't help timelineservice.app_flow table
[ https://issues.apache.org/jira/browse/YARN-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-9303: - Labels: atsv2 atsv2-hbase (was: atsv2) > Username splits won't help timelineservice.app_flow table > - > > Key: YARN-9303 > URL: https://issues.apache.org/jira/browse/YARN-9303 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.1.2 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: atsv2, atsv2-hbase > Attachments: Only_Last_Region_Used.png, YARN-9303-001.patch > > > timelineservice.app_flow hbase table uses pre split logic based on username > whereas the rowkeys starts with inverted timestamp (Long.MAX_VALUE - ts). All > data will go to the last region and remaining regions will never be inserted. > Need to choose right split or use auto-split. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9303) Username splits won't help timelineservice.app_flow table
[ https://issues.apache.org/jira/browse/YARN-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-9303: - Labels: atsv2 (was: ) > Username splits won't help timelineservice.app_flow table > - > > Key: YARN-9303 > URL: https://issues.apache.org/jira/browse/YARN-9303 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.1.2 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: atsv2 > Attachments: Only_Last_Region_Used.png, YARN-9303-001.patch > > > timelineservice.app_flow hbase table uses pre split logic based on username > whereas the rowkeys starts with inverted timestamp (Long.MAX_VALUE - ts). All > data will go to the last region and remaining regions will never be inserted. > Need to choose right split or use auto-split. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9303) Username splits won't help timelineservice.app_flow table
[ https://issues.apache.org/jira/browse/YARN-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-9303: - Fix Version/s: 3.3.0 > Username splits won't help timelineservice.app_flow table > - > > Key: YARN-9303 > URL: https://issues.apache.org/jira/browse/YARN-9303 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.1.2 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: atsv2, atsv2-hbase > Fix For: 3.3.0 > > Attachments: Only_Last_Region_Used.png, YARN-9303-001.patch > > > timelineservice.app_flow hbase table uses pre split logic based on username > whereas the rowkeys starts with inverted timestamp (Long.MAX_VALUE - ts). All > data will go to the last region and remaining regions will never be inserted. > Need to choose right split or use auto-split. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9335) [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls
[
https://issues.apache.org/jira/browse/YARN-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809528#comment-16809528
]
Vrushali C commented on YARN-9335:
--
Hi Abhishek,
Could you check applying this patch as well? It seems to not work for me. Do
you see anything incorrect in my command below:
{code}
[tw-mbp13-channapattan hadoop (trunk)]$ git apply -p0
~/Downloads/YARN-9335.003.patch
error:
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java:
No such file or directory
error:
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml:
No such file or directory
error:
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollector.java:
No such file or directory
error:
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestTimelineCollector.java:
No such file or directory
[tw-mbp13-channapattan hadoop (trunk)]$ git apply -p1
~/Downloads/YARN-9335.003.patch
error: patch failed:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollector.java:221
error:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollector.java:
patch does not apply
[tw-mbp13-channapattan hadoop (trunk)]$
{code}
> [atsv2] Restrict the number of elements held in timeline collector when
> backend is unreachable for async calls
> --
>
> Key: YARN-9335
> URL: https://issues.apache.org/jira/browse/YARN-9335
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Vrushali C
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-9335.001.patch, YARN-9335.002.patch,
> YARN-9335.003.patch
>
>
> For ATSv2 , if the backend is unreachable, the number/size of data held in
> timeline collector's memory increases significantly. This is not good for the
> NM memory.
> Filing jira to set a limit on how many/much should be retained by the
> timeline collector in memory in case the backend is not reachable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9382) Publish container killed, paused and resumed events to ATSv2.
[
https://issues.apache.org/jira/browse/YARN-9382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809527#comment-16809527
]
Vrushali C commented on YARN-9382:
--
Hi Abhishek,
I am somehow not able to apply the patch (with p0 or p1). Can you check?
{code}
[tw-mbp13-channapattan hadoop (trunk)]$ git apply -p0 -v
~/Downloads/YARN-9382.002.patch
Checking patch
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ContainerMetricsConstants.java
=>
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ContainerMetricsConstants.java...
error:
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ContainerMetricsConstants.java:
No such file or directory
Checking patch
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
=>
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java...
error:
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java:
No such file or directory
Checking patch
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/TestNMTimelinePublisher.java
=>
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/TestNMTimelinePublisher.java...
error:
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/TestNMTimelinePublisher.java:
No such file or directory
[tw-mbp13-channapattan hadoop (trunk)]$
{code}
{code}
[tw-mbp13-channapattan hadoop (trunk)]$ git apply -p1 -v
~/Downloads/YARN-9382.002.patch
Checking patch
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ContainerMetricsConstants.java...
Checking patch
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java...
Hunk #2 succeeded at 255 (offset -7 lines).
error: while searching for:
case INIT_CONTAINER:
publishContainerCreatedEvent(event);
break;
default:
if (LOG.isDebugEnabled()) {
LOG.debug(event.getType()
error: patch failed:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java:402
error:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java:
patch does not apply
Checking patch
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/TestNMTimelinePublisher.java...
error: while searching for:
import org.apache.hadoop.yarn.server.nodemanager.Context;
import
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationContainerFinishedEvent;
import
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
import org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree;
import org.junit.Assert;
import org.junit.Test;
error: patch failed:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/TestNMTimelinePublisher.java:45
error:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/TestNMTimelinePublisher.java:
patch does not apply
[tw-mbp13-channapattan hadoop (trunk)]$
{code}
> Publish container killed, paused and resumed events to ATSv2.
> -
>
> Key: YARN-9382
> URL: https://issues.apache.org/jira/browse/YARN-9382
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-9382.001.patch, YARN-9382.002.patch
>
>
> There are some events missing in container lifecycle. We need to add support
> for adding events for when container gets killed, paused and resumed.
--
This message was sent by Atlassian JIRA
[jira] [Commented] (YARN-9403) GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table
[
https://issues.apache.org/jira/browse/YARN-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809526#comment-16809526
]
Vrushali C commented on YARN-9403:
--
I am not sure I understand the issue correctly. For YARN_APPLICATIOn entities,
they are being written to the application table, no? If so, why do we need to
go to the entities table? If there any information missing in the response that
was expected.
> GET /apps/{appid}/entities/YARN_APPLICATION accesses application table
> instead of entity table
> --
>
> Key: YARN-9403
> URL: https://issues.apache.org/jira/browse/YARN-9403
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9403-001.patch, YARN-9403-002.patch,
> YARN-9403-003.patch, YARN-9403-004.patch
>
>
> {noformat}"GET /apps/{appid}/entities/YARN_APPLICATION"{noformat} accesses
> application table instead of entity table. As per the doc, With this API, you
> can query generic entities identified by cluster ID, application ID and
> per-framework entity type. But it also provides all the apps when entityType
> is set to YARN_APPLICATION. It should only access Entity Table through
> {{GenericEntityReader}}.
> Wrong Output: With YARN_APPLICATION entityType, all applications listed from
> application tables.
> {code}
> [hbase@yarn-ats-3 centos]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/YARN_APPLICATION?user.name=hbase=hbase=word%20count;
> | jq .
> [
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553258922721,
> "idprefix": 0,
> "isrelatedto": {},
> "relatesto": {},
> "info": {
> "UID": "ats!application_1553258815132_0002",
> "FROM_ID": "ats!hbase!word
> count!1553258922721!application_1553258815132_0002"
> },
> "configs": {},
> "type": "YARN_APPLICATION",
> "id": "application_1553258815132_0002"
> },
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553258825918,
> "idprefix": 0,
> "isrelatedto": {},
> "relatesto": {},
> "info": {
> "UID": "ats!application_1553258815132_0001",
> "FROM_ID": "ats!hbase!word
> count!1553258825918!application_1553258815132_0001"
> },
> "configs": {},
> "type": "YARN_APPLICATION",
> "id": "application_1553258815132_0001"
> }
> ]
> {code}
> Right Output: With correct entity type (MAPREDUCE_JOB) it accesses entity
> table for given applicationId and entityType.
> {code}
> [hbase@yarn-ats-3 centos]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/MAPREDUCE_JOB?user.name=hbase=hbase=word%20count;
> | jq .
> [
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553258926667,
> "idprefix": 0,
> "isrelatedto": {},
> "relatesto": {},
> "info": {
> "UID":
> "ats!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002",
> "FROM_ID": "ats!hbase!word
> count!1553258922721!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002"
> },
> "configs": {},
> "type": "MAPREDUCE_JOB",
> "id": "job_1553258815132_0002"
> }
> ]
> {code}
> Flow Activity and Flow Run tables can also be accessed using similar way.
> {code}
> GET /apps/{appid}/entities/YARN_FLOW_ACTIVITY
> GET /apps/{appid}/entities/YARN_FLOW_RUN
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9408) @Path("/apps/{appid}/appattempts") error message misleads
[
https://issues.apache.org/jira/browse/YARN-9408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809525#comment-16809525
]
Vrushali C commented on YARN-9408:
--
Hmm, so I am trying understand this error. Looks like it may be thrown at this
line
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/reader/AbstractTimelineStorageReader.java#L85
It's because there result set was empty/null.
Looking at the code, it is trying to look up the flow context for this app id
and it does not find anything. I am wondering if catching all
NotFoundExceptions is a good idea. Perhaps we can add to the exception message
and enhance it rather than printing out a completely new message.
> @Path("/apps/{appid}/appattempts") error message misleads
> -
>
> Key: YARN-9408
> URL: https://issues.apache.org/jira/browse/YARN-9408
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9408-001.patch, YARN-9408-002.patch
>
>
> {code} @Path("/apps/{appid}/appattempts") {code} error message is misleading.
> NotFoundException "Unable to find the context flow name, and flow run id, and
> user id" is displayed while app attempts is looked.
> {code}
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0004/appattempts?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats,
> appId=application_1553258815132_0004",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/clusters/ats/apps/application_1553258815132_0005/appattempts?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats,
> appId=application_1553258815132_0005",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/clusters/ats1/apps/application_1553258815132_0001/containers/container_e14_1553258815132_0001_01_01?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats1,
> appId=application_1553258815132_0001",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/clusters/ats1/apps/application_1553258815132_0001/appattempts/appattempt_1553258815132_0001_01/containers?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats1,
> appId=application_1553258815132_0001",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9303) Username splits won't help timelineservice.app_flow table
[ https://issues.apache.org/jira/browse/YARN-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809392#comment-16809392 ] Vrushali C commented on YARN-9303: -- +1 to patch v1. I am reviewing the other patch. But this one is correct, will commit shortly > Username splits won't help timelineservice.app_flow table > - > > Key: YARN-9303 > URL: https://issues.apache.org/jira/browse/YARN-9303 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.1.2 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: Only_Last_Region_Used.png, YARN-9303-001.patch > > > timelineservice.app_flow hbase table uses pre split logic based on username > whereas the rowkeys starts with inverted timestamp (Long.MAX_VALUE - ts). All > data will go to the last region and remaining regions will never be inserted. > Need to choose right split or use auto-split. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9373) HBaseTimelineSchemaCreator has to allow user to configure pre-splits
[ https://issues.apache.org/jira/browse/YARN-9373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809390#comment-16809390 ] Vrushali C commented on YARN-9373: -- Thanks Prabhu, overall patch v2 looks good. I want to look at it a bit more in detail today. Will update the jira with comments, else will commit it. > HBaseTimelineSchemaCreator has to allow user to configure pre-splits > > > Key: YARN-9373 > URL: https://issues.apache.org/jira/browse/YARN-9373 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: Configurable_PreSplits.png, YARN-9373-001.patch, > YARN-9373-002.patch > > > Most of the TimelineService HBase tables is set with username splits which is > based on lowercase alphabet (a,ad,an,b,ca). This won't help if the rowkey > starts with either number or uppercase alphabet. We need to allow user to > configure based upon their data. For example, say a user has configured the > yarn.resourcemanager.cluster-id to be ATS or 123, then the splits can be > configured as A,B,C,,, or 100,200,300,,, -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9382) Publish container killed, paused and resumed events to ATSv2.
[ https://issues.apache.org/jira/browse/YARN-9382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809387#comment-16809387 ] Vrushali C commented on YARN-9382: -- thanks Abhishek, patch v2 looks good. Will commit it shortly > Publish container killed, paused and resumed events to ATSv2. > - > > Key: YARN-9382 > URL: https://issues.apache.org/jira/browse/YARN-9382 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9382.001.patch, YARN-9382.002.patch > > > There are some events missing in container lifecycle. We need to add support > for adding events for when container gets killed, paused and resumed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9335) [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls
[ https://issues.apache.org/jira/browse/YARN-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809385#comment-16809385 ] Vrushali C commented on YARN-9335: -- Thanks for the patch v3 Abhishek, lgtm. Will commit shortly > [atsv2] Restrict the number of elements held in timeline collector when > backend is unreachable for async calls > -- > > Key: YARN-9335 > URL: https://issues.apache.org/jira/browse/YARN-9335 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vrushali C >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9335.001.patch, YARN-9335.002.patch, > YARN-9335.003.patch > > > For ATSv2 , if the backend is unreachable, the number/size of data held in > timeline collector's memory increases significantly. This is not good for the > NM memory. > Filing jira to set a limit on how many/much should be retained by the > timeline collector in memory in case the backend is not reachable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9382) Publish container killed, paused and resumed events to ATSv2.
[
https://issues.apache.org/jira/browse/YARN-9382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798744#comment-16798744
]
Vrushali C commented on YARN-9382:
--
Thanks Abhishek! Patch v1 LGTM.
Could you update the checkstyle warnings?
{code}
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java:277:
entity.setIdPrefix(TimelineServiceHelper.invertLong(containerStartTime));: Line
is longer than 80 characters (found 81). [LineLength]
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/TestNMTimelinePublisher.java:27:import
java.util.HashMap;:8: Unused import - java.util.HashMap. [UnusedImports]
{code}
Also, I was wondering, if we can call it
"publishContainerLifeCycleGenericEvent" instead of
"publishContainerGenericEvent" ? Or perhaps Generic Event is fine too.
One other question I had was, should KILLED event have anything that FINISHED
event (publishContainerFinishedEvent) has too? Like the diagnostics, exit info
etc?
> Publish container killed, paused and resumed events to ATSv2.
> -
>
> Key: YARN-9382
> URL: https://issues.apache.org/jira/browse/YARN-9382
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-9382.001.patch
>
>
> There are some events missing in container lifecycle. We need to add support
> for adding events for when container gets killed, paused and resumed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9366) Make logs in TimelineClient implementation specific to application
[ https://issues.apache.org/jira/browse/YARN-9366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798740#comment-16798740 ] Vrushali C commented on YARN-9366: -- Thanks [~prabham] , Patch v1 LGTM. Will wait for jenkins for overall checks and then commit. > Make logs in TimelineClient implementation specific to application > --- > > Key: YARN-9366 > URL: https://issues.apache.org/jira/browse/YARN-9366 > Project: Hadoop YARN > Issue Type: Improvement > Components: ATSv2 >Reporter: Prabha Manepalli >Assignee: Prabha Manepalli >Priority: Minor > Attachments: YARN-9366.v1.patch > > > For every container launched on a NM node, a timeline client is created to > publish entities to the corresponding application's timeline collector. And > there would be multiple timeline clients running at the same time. Current > implementation of timeline client logs are insufficient to isolate publishing > problems related to one application. Hence, creating this Jira to improvise > the logs in TimelineV2ClientImpl. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9395) Short Names for repeated Hbase Column names
[
https://issues.apache.org/jira/browse/YARN-9395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798738#comment-16798738
]
Vrushali C commented on YARN-9395:
--
Very good jira Prabhu. I agree that majority of the counter names are going to
be repeated over and over again across jobs. Please do give it some thought and
let's discuss about potential solutions.
With Phoenix, since they have a predefined schema for a table, it is an option
to have a mapping for a column name to a number.
> Short Names for repeated Hbase Column names
> ---
>
> Key: YARN-9395
> URL: https://issues.apache.org/jira/browse/YARN-9395
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>
> Currently ATS HBase tables stores the config name / metric name as column
> names which are long. This repeats for all the rows and consumes lot of
> storage space. And we have seen Customers Hbase Tables already consumes more
> than 1.5 TB in few days
> {code}
> Example Configs:
> c:yarn.timeline-service.webapp.rest-csrf.methods-to-ignore
> c:yarn.timeline-service.entity-group-fs-store.active-dir
> c:yarn.scheduler.configuration.zk-store.parent-path
> Example Metrics:
> m:REDUCE:org.apache.hadoop.mapreduce.FileSystemCounter:HDFS_READ_OPS
> m:REDUCE:org.apache.hadoop.mapreduce.TaskCounter:COMBINE_INPUT_RECORDS
> m:REDUCE:org.apache.hadoop.mapreduce.TaskCounter:PHYSICAL_MEMORY_BYTES
> {code}
> We need to use short column names as per Hbase Best Practice -
> http://moi.vonos.net/bigdata/avro-hbase-colnames/ But the challenge is ATS
> does not know the column names until the rows get inserted. We can provide a
> mapping file to map the repeated configs / metrics / info from different
> applications to unique numbers which customers can configure upfront to save
> the storage space. Similar to what Phoenix does
> https://blogs.apache.org/phoenix/entry/column-mapping-and-immutable-data
> https://phoenix.apache.org/columnencoding.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2
[
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791328#comment-16791328
]
Vrushali C commented on YARN-9016:
--
We discussed this during our atsv2 community call. Our collective understanding
is that, as per the current patch, this patch would add in the azure-documentdb
jar in the NM classpath when timeline service v2 is used. The other
dependencies azure-documentdb requires are already provided, so no additional
new jars should be coming into the NM classpath.
Sushil is going to file a separate jira (to be fixed later, independently of
this jira) that will add in a profile based compilation for document store.
We are waiting on the jenkins build which has been running 4+ hours. Once the
build finishes, we can commit this patch.
> DocumentStore as a backend for ATSv2
>
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch, YARN-9016.002.patch,
> YARN-9016.003.patch, YARN-9016.004.patch
>
>
> h1. Document Store for ATSv2
> The Document Store for ATSv2 is a framework for plugging in
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB ,
> MongoDB, ElasticSearch etc.
> * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch,
> MongoDB etc by just adding new configurations properties and writing Document
> Store reader and writer clients.
> * Currently has support for CosmosDB.
> * All writes are Async and buffered, latest document would be flushed to the
> store either if the document buffer gets full or periodically at every flush
> interval in background without adding any additional latency to the running
> jobs..
> * All the REST API's of Timeline Reader Server are supported.
> h4.
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.writer.class
>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
>
>
>yarn.timeline-service.reader.class
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
>
>
>yarn.timeline-service.document-store.db-name
>YOUR_DATABASE_NAME
> {code}
> h3. *Creating DB and Collections for storing documents*
> The following config needs to be set inside
> *yarn-site.xml* for creating the database and collections for storing
> documents.
> {code:java}
>
>
> yarn.timeline-service.schema-creator.class
>
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> Running the schema creator tool to create the necessary
> collections.
> {code:java}
> bin/hadoop
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
> h3. *Azure CosmosDB*
> To use Azure CosmosDB as a DocumentStore for ATSv2, the additional
> properties under *yarn-site.xml* is required..
> {code:java}
>
>
>yarn.timeline-service.document-store-type
>COSMOS_DB
>
>
>yarn.timeline-service.document-store.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
>
>
>yarn.timeline-service.document-store.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
>
> {code}
>
> *Testing locally*
> In order to test the Azure CosmosDB as a DocumentStore
> locally, install the emulator from
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and
> start it locally. Set the endpoint and master key under *yarn-site.xml* as
> mentioned above and run any example job like DistributedShell etc. Later you
> can check the data explorer UI of Azure CosmosDB locally to query the
> documents or even launch the *TimelineReader* locally to fetch/query the data
> from REST API's.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2
[
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791334#comment-16791334
]
Vrushali C commented on YARN-9016:
--
Two of the tests are failing due to bind exceptions
{code}
[ERROR]
testTimelineServiceEventPublishingV1V2Enabled(org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher)
Time elapsed: 0.088 s <<< ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.BindException:
Problem binding to [0.0.0.0:10200] java.net.BindException: Address already in
use; For more details see: http://wiki.apache.org/hadoop/BindException
at
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:139)
at
org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:66)
at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:55)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService.serviceStart(ApplicationHistoryClientService.java:94)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceStart(ApplicationHistoryServer.java:120)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.testSetup(TestCombinedSystemMetricsPublisher.java:123)
at
org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.runTest(TestCombinedSystemMetricsPublisher.java:242)
at
org.apache.hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV1V2Enabled(TestCombinedSystemMetricsPublisher.java:252)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.BindException: Problem binding to [0.0.0.0:10200]
java.net.BindException: Address already in use; For more details see:
http://wiki.apache.org/hadoop/BindException
{code}
The
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor)
is failing due to:
{code}
Caused by: org.apache.hadoop.yarn.exceptions.YarnException:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: amrmToken from UAM SC-1
should be null here
at
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.FederationInterceptor.allocate(FederationInterceptor.java:719)
at
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor.getContainersAndAssert(TestFederationInterceptor.java:219)
at
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor.access$400(TestFederationInterceptor.java:89)
at
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor$2.run(TestFederationInterceptor.java:407)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
... 28 more
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: amrmToken
from UAM SC-1 should be null here
at
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.FederationInterceptor.mergeAllocateResponse(FederationInterceptor.java:1418)
at
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.FederationInterceptor.mergeAllocateResponses(FederationInterceptor.java:1348)
at
org.apache.hadoop.yarn.server.nodemanager.amrmproxy.FederationInterceptor.allocate(FederationInterceptor.java:695)
... 34 more
{code}
All of these seem to be unrelated to the patch.
So +1 for patch v004.
I will give it a
[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2
[
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791180#comment-16791180
]
Vrushali C commented on YARN-9016:
--
Also, as noted in YARN-7190 for the hbase jars, we need to ensure that the
document store related jars are not in the user classpath.
> DocumentStore as a backend for ATSv2
>
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch, YARN-9016.002.patch,
> YARN-9016.003.patch, YARN-9016.004.patch
>
>
> h1. Document Store for ATSv2
> The Document Store for ATSv2 is a framework for plugging in
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB ,
> MongoDB, ElasticSearch etc.
> * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch,
> MongoDB etc by just adding new configurations properties and writing Document
> Store reader and writer clients.
> * Currently has support for CosmosDB.
> * All writes are Async and buffered, latest document would be flushed to the
> store either if the document buffer gets full or periodically at every flush
> interval in background without adding any additional latency to the running
> jobs..
> * All the REST API's of Timeline Reader Server are supported.
> h4.
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.writer.class/name>
>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
>
>
>yarn.timeline-service.reader.class/name>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
>
>
>yarn.timeline-service.documentstore.db-name
>YOUR_DATABASE_NAME
> {code}
> h3. *Creating DB and Collections for storing documents*
> This is similar to HBase *TimelineSchemaCreator* the
> following command needs to be executed once for setting up the database and
> collections for storing documents.
> {code:java}
> hadoop
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3. *Azure CosmosDB*
> To use Azure CosmosDB as a DocumentStore for ATSv2, the additional
> properties under *yarn-site.xml* is required..
> {code:java}
>
>
>yarn.timeline-service.store-type
>COSMOS_DB
>
>
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
>
>
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
>
> {code}
>
> *Testing locally*
> In order to test the Azure CosmosDB as a DocumentStore
> locally, install the emulator from
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and
> start it locally. Set the endpoint and master key under *yarn-site.xml* as
> mentioned above and run any example job like DistributedShell etc. Later you
> can check the data explorer UI of Azure CosmosDB locally to query the
> documents or even launch the *TimelineReader* locally and fetch/query the
> data from REST API's.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2
[
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791174#comment-16791174
]
Vrushali C commented on YARN-9016:
--
Thanks for updating the patch Sushil!
Patch 004 LGTM in general but I have a few things to say.
- I believe, if the settings are not enabled for DocumentStore, then nothing in
the new code will get executed. Also, I believe it's not enabled by default.
- My only question is about the new jars that may be coming into the classpath.
For trunk this should still be ok, this is just more of a knowledge question.
Also, I would like to confirm that the unit tests pass to make sure , I will
retrigger jenkins now.
> DocumentStore as a backend for ATSv2
>
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch, YARN-9016.002.patch,
> YARN-9016.003.patch, YARN-9016.004.patch
>
>
> h1. Document Store for ATSv2
> The Document Store for ATSv2 is a framework for plugging in
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB ,
> MongoDB, ElasticSearch etc.
> * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch,
> MongoDB etc by just adding new configurations properties and writing Document
> Store reader and writer clients.
> * Currently has support for CosmosDB.
> * All writes are Async and buffered, latest document would be flushed to the
> store either if the document buffer gets full or periodically at every flush
> interval in background without adding any additional latency to the running
> jobs..
> * All the REST API's of Timeline Reader Server are supported.
> h4.
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.writer.class/name>
>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
>
>
>yarn.timeline-service.reader.class/name>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
>
>
>yarn.timeline-service.documentstore.db-name
>YOUR_DATABASE_NAME
> {code}
> h3. *Creating DB and Collections for storing documents*
> This is similar to HBase *TimelineSchemaCreator* the
> following command needs to be executed once for setting up the database and
> collections for storing documents.
> {code:java}
> hadoop
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3. *Azure CosmosDB*
> To use Azure CosmosDB as a DocumentStore for ATSv2, the additional
> properties under *yarn-site.xml* is required..
> {code:java}
>
>
>yarn.timeline-service.store-type
>COSMOS_DB
>
>
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
>
>
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
>
> {code}
>
> *Testing locally*
> In order to test the Azure CosmosDB as a DocumentStore
> locally, install the emulator from
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and
> start it locally. Set the endpoint and master key under *yarn-site.xml* as
> mentioned above and run any example job like DistributedShell etc. Later you
> can check the data explorer UI of Azure CosmosDB locally to query the
> documents or even launch the *TimelineReader* locally and fetch/query the
> data from REST API's.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9338) Timeline related testcases are failing
[
https://issues.apache.org/jira/browse/YARN-9338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789802#comment-16789802
]
Vrushali C commented on YARN-9338:
--
Thanks for the patch [~abmodi] ! Patch 004 LGTM. I assume the unit tests are
passing given the jenkins build is green.
Will commit it later today.
> Timeline related testcases are failing
> --
>
> Key: YARN-9338
> URL: https://issues.apache.org/jira/browse/YARN-9338
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Prabhu Joseph
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-9338.001.patch, YARN-9338.002.patch,
> YARN-9338.003.patch, YARN-9338.004.patch
>
>
> Timeline related testcases are failing
> {code}
> [ERROR] Failures:
> [ERROR]
> TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV2Enabled:262->runTest:245->validateV2:382->verifyEntity:417
> Expected 2 events to be published expected:<2> but was:<1>
> [ERROR]
> TestSystemMetricsPublisherForV2.testPublishAppAttemptMetrics:259->verifyEntity:332
> Expected 2 events to be published expected:<2> but was:<1>
> [ERROR]
> TestSystemMetricsPublisherForV2.testPublishApplicationMetrics:224->verifyEntity:332
> Expected 4 events to be published expected:<4> but was:<1>
> [ERROR]
> TestSystemMetricsPublisherForV2.testPublishContainerMetrics:291->verifyEntity:332
> Expected 2 events to be published expected:<2> but was:<1>
> [ERROR] Errors:
> [ERROR]
> TestCombinedSystemMetricsPublisher.testTimelineServiceEventPublishingV1V2Enabled:252->runTest:242->testSetup:123
> » YarnRuntime
> [ERROR] Failures:
> [ERROR]
> TestTimelineAuthFilterForV2.testPutTimelineEntities:343->access$000:87->publishAndVerifyEntity:307
> expected:<1> but was:<2>
> [ERROR]
> TestTimelineAuthFilterForV2.testPutTimelineEntities:352->publishWithRetries:320->publishAndVerifyEntity:307
> expected:<1> but was:<2>
> [ERROR]
> TestTimelineAuthFilterForV2.testPutTimelineEntities:352->publishWithRetries:320->publishAndVerifyEntity:307
> expected:<1> but was:<2>
> [ERROR]
> TestTimelineAuthFilterForV2.testPutTimelineEntities:343->access$000:87->publishAndVerifyEntity:307
> expected:<1> but was:<2>
> [INFO]
> [ERROR] Failures:
> [ERROR]
> TestDistributedShell.testDSShellWithoutDomainV2:313->testDSShell:317->testDSShell:458->checkTimelineV2:557->verifyEntityForTimelineV2:710
> Unexpected number of DS_APP_ATTEMPT_START event published. expected:<1> but
> was:<0>
> [ERROR]
> TestDistributedShell.testDSShellWithoutDomainV2CustomizedFlow:329->testDSShell:458->checkTimelineV2:557->verifyEntityForTimelineV2:710
> Unexpected number of DS_APP_ATTEMPT_START event published. expected:<1> but
> was:<0>
> [ERROR]
> TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow:323->testDSShell:458->checkTimelineV2:557->verifyEntityForTimelineV2:710
> Unexpected number of DS_APP_ATTEMPT_START event published. expected:<1> but
> was:<0>
> [ERROR] Failures:
> [ERROR]
> TestMRTimelineEventHandling.testMRNewTimelineServiceEventHandling:240->checkNewTimelineEvent:304->verifyEntity:462
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9335) [atsv2] Restrict the number of elements held in NM timeline collector when backend is unreachable for asycn calls
[ https://issues.apache.org/jira/browse/YARN-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787605#comment-16787605 ] Vrushali C commented on YARN-9335: -- As discussed in the community call today, we can tackle the solution for async writes in this jira. We can have another jira to handle the situation with sync writes. > [atsv2] Restrict the number of elements held in NM timeline collector when > backend is unreachable for asycn calls > - > > Key: YARN-9335 > URL: https://issues.apache.org/jira/browse/YARN-9335 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vrushali C >Assignee: Abhishek Modi >Priority: Major > > For ATSv2 , if the backend is unreachable, the number/size of data held in > timeline collector's memory increases significantly. This is not good for the > NM memory. > Filing jira to set a limit on how many/much should be retained by the > timeline collector in memory in case the backend is not reachable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8499) ATS v2 should handle connection issues in general for all storages
[ https://issues.apache.org/jira/browse/YARN-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-8499: - Labels: atsv2 (was: ) > ATS v2 should handle connection issues in general for all storages > -- > > Key: YARN-8499 > URL: https://issues.apache.org/jira/browse/YARN-8499 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sunil Govindan >Assignee: Prabhu Joseph >Priority: Major > Labels: atsv2 > Attachments: YARN-8499-001.patch, YARN-8499-002.patch, > YARN-8499-003.patch, YARN-8499-004.patch > > > Post YARN-8302, Hbase connection issues are handled in ATSv2. However this > could be made general by introducing an api in storage interface and > implementing in each of the storage as per the store semantics. > > cc [~rohithsharma] [~vinodkv] [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8549) Adding a NoOp timeline writer and reader plugin classes for ATSv2
[ https://issues.apache.org/jira/browse/YARN-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787595#comment-16787595 ] Vrushali C commented on YARN-8549: -- Thanks for the branch-2 patch Prabha! I will commit it now. > Adding a NoOp timeline writer and reader plugin classes for ATSv2 > - > > Key: YARN-8549 > URL: https://issues.apache.org/jira/browse/YARN-8549 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2, timelineclient, timelineserver >Reporter: Prabha Manepalli >Assignee: Prabha Manepalli >Priority: Minor > Attachments: YARN-8549-branch-2.001.patch, YARN-8549.v1.patch, > YARN-8549.v2.patch, YARN-8549.v4.patch, YARN-8549.v5.patch > > > Stub implementation for TimeLineReader and TimeLineWriter classes. > These are useful for functional testing of writer and reader path for ATSv2 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8302) ATS v2 should handle HBase connection issue properly
[
https://issues.apache.org/jira/browse/YARN-8302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-8302:
-
Labels: atsv2 (was: )
> ATS v2 should handle HBase connection issue properly
>
>
> Key: YARN-8302
> URL: https://issues.apache.org/jira/browse/YARN-8302
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Assignee: Billie Rinaldi
>Priority: Major
> Labels: atsv2
> Fix For: 3.2.0, 3.1.1
>
> Attachments: YARN-8302.1.patch, YARN-8302.2.patch, YARN-8302.3.patch
>
>
> ATS v2 call times out with below error when it can't connect to HBase
> instance.
> {code}
> bash-4.2$ curl -i -k -s -1 -H 'Content-Type: application/json' -H 'Accept:
> application/json' --max-time 5 --negotiate -u :
> 'https://xxx:8199/ws/v2/timeline/apps/application_1526357251888_0022/entities/YARN_CONTAINER?fields=ALL&_=1526425686092'
> curl: (28) Operation timed out after 5002 milliseconds with 0 bytes received
> {code}
> {code:title=ATS log}
> 2018-05-15 23:10:03,623 INFO client.RpcRetryingCallerImpl
> (RpcRetryingCallerImpl.java:callWithRetries(134)) - Call exception, tries=7,
> retries=7, started=8165 ms ago, cancelled=false, msg=Call to xxx/xxx:17020
> failed on connection exception:
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
> Connection refused: xxx/xxx:17020, details=row
> 'prod.timelineservice.app_flow,
> ,99' on table 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=xxx,17020,1526348294182, seqNum=-1
> 2018-05-15 23:10:13,651 INFO client.RpcRetryingCallerImpl
> (RpcRetryingCallerImpl.java:callWithRetries(134)) - Call exception, tries=8,
> retries=8, started=18192 ms ago, cancelled=false, msg=Call to xxx/xxx:17020
> failed on connection exception:
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
> Connection refused: xxx/xxx:17020, details=row
> 'prod.timelineservice.app_flow,
> ,99' on table 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=xxx,17020,1526348294182, seqNum=-1
> 2018-05-15 23:10:23,730 INFO client.RpcRetryingCallerImpl
> (RpcRetryingCallerImpl.java:callWithRetries(134)) - Call exception, tries=9,
> retries=9, started=28272 ms ago, cancelled=false, msg=Call to xxx/xxx:17020
> failed on connection exception:
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
> Connection refused: xxx/xxx:17020, details=row
> 'prod.timelineservice.app_flow,
> ,99' on table 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=xxx,17020,1526348294182, seqNum=-1
> 2018-05-15 23:10:33,788 INFO client.RpcRetryingCallerImpl
> (RpcRetryingCallerImpl.java:callWithRetries(134)) - Call exception, tries=10,
> retries=10, started=38330 ms ago, cancelled=false, msg=Call to xxx/xxx:17020
> failed on connection exception:
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
> Connection refused: xxx/xxx:17020, details=row
> 'prod.timelineservice.app_flow,
> ,99' on table 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=xxx,17020,1526348294182, seqNum=-1{code}
> There are two issues here.
> 1) Check why ATS can't connect to HBase
> 2) In case of connection error, ATS call should not get timeout. It should
> fail with proper error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8218) Add application launch time to ATSV1
[ https://issues.apache.org/jira/browse/YARN-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787003#comment-16787003 ] Vrushali C commented on YARN-8218: -- https://builds.apache.org/job/Hadoop-trunk-Commit/16152/ was successful > Add application launch time to ATSV1 > > > Key: YARN-8218 > URL: https://issues.apache.org/jira/browse/YARN-8218 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Kanwaljeet Sachdev >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-8218.001.patch > > > YARN-7088 publishes application launch time to RMStore and also adds it to > the YARN UI. It would be a nice enhancement to have the launchTime event > published into the Application history server as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8218) Add application launch time to ATSV1
[ https://issues.apache.org/jira/browse/YARN-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786449#comment-16786449 ] Vrushali C commented on YARN-8218: -- Retriggered the build. It had failed at hadoop-common. > Add application launch time to ATSV1 > > > Key: YARN-8218 > URL: https://issues.apache.org/jira/browse/YARN-8218 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Kanwaljeet Sachdev >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-8218.001.patch > > > YARN-7088 publishes application launch time to RMStore and also adds it to > the YARN UI. It would be a nice enhancement to have the launchTime event > published into the Application history server as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8218) Add application launch time to ATSV1
[ https://issues.apache.org/jira/browse/YARN-8218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779894#comment-16779894 ] Vrushali C commented on YARN-8218: -- Thanks Abhishek! Patch 001 LGTM. > Add application launch time to ATSV1 > > > Key: YARN-8218 > URL: https://issues.apache.org/jira/browse/YARN-8218 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Kanwaljeet Sachdev >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8218.001.patch > > > YARN-7088 publishes application launch time to RMStore and also adds it to > the YARN UI. It would be a nice enhancement to have the launchTime event > published into the Application history server as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3841) [Storage implementation] Adding retry semantics to HDFS backing storage
[
https://issues.apache.org/jira/browse/YARN-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-3841:
-
Labels: YARN-5355 atsv2 (was: YARN-5355)
> [Storage implementation] Adding retry semantics to HDFS backing storage
> ---
>
> Key: YARN-3841
> URL: https://issues.apache.org/jira/browse/YARN-3841
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Tsuyoshi Ozawa
>Assignee: Abhishek Modi
>Priority: Major
> Labels: YARN-5355, atsv2
> Fix For: 3.3.0
>
> Attachments: YARN-3841-YARN-7055.002.patch, YARN-3841.001.patch,
> YARN-3841.002.patch, YARN-3841.003.patch, YARN-3841.004.patch,
> YARN-3841.005.patch, YARN-3841.006.patch, YARN-3841.007.patch,
> YARN-3841.008.patch, YARN-3841.009.patch
>
>
> HDFS backing storage is useful for following scenarios.
> 1. For Hadoop clusters which don't run HBase.
> 2. For fallback from HBase when HBase cluster is temporary unavailable.
> Quoting ATS design document of YARN-2928:
> {quote}
> In the case the HBase
> storage is not available, the plugin should buffer the writes temporarily
> (e.g. HDFS), and flush
> them once the storage comes back online. Reading and writing to hdfs as the
> the backup storage
> could potentially use the HDFS writer plugin unless the complexity of
> generalizing the HDFS
> writer plugin for this purpose exceeds the benefits of reusing it here.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2
[
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779865#comment-16779865
]
Vrushali C commented on YARN-9150:
--
Unable to apply patch to branch-2, I have reached out to Sushil.
> Making TimelineSchemaCreator support different backends for Timeline Schema
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150-branch-2.001.patch,
> YARN-9150-branch-2.002.patch, YARN-9150-branch-2.003.patch,
> YARN-9150-branch-2.patch, YARN-9150.001.patch, YARN-9150.002.patch,
> jenkins_build.png
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for
> creating Timeline Schema's only for HBase, Hence creating this JIRA for
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
> Add the following property in *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.schema-creator.class
>
> YOUR_TIMELINE_SCHEMA_CREATOR_CLASS
>
> {code}
> The Command needed to run the TimelineSchemaCreator need not be changed
> i.e the below existing command can be used irrespective of the backend
> configured.
> {code:java}
> bin/hadoop
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
> -create
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3841) [Storage implementation] Adding retry semantics to HDFS backing storage
[
https://issues.apache.org/jira/browse/YARN-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779857#comment-16779857
]
Vrushali C commented on YARN-3841:
--
Thanks for the patch v009 [~abmodi]! I have committed that to trunk as part of
https://github.com/apache/hadoop/commit/ea3cdc60b39d96702c0bce292829914c25bc0d8e
Thanks to [~zjshen] , [~gtCarrera9] [~jrottinghuis] [~ozawa] for the discussion
around this.
> [Storage implementation] Adding retry semantics to HDFS backing storage
> ---
>
> Key: YARN-3841
> URL: https://issues.apache.org/jira/browse/YARN-3841
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Tsuyoshi Ozawa
>Assignee: Abhishek Modi
>Priority: Major
> Labels: YARN-5355
> Attachments: YARN-3841-YARN-7055.002.patch, YARN-3841.001.patch,
> YARN-3841.002.patch, YARN-3841.003.patch, YARN-3841.004.patch,
> YARN-3841.005.patch, YARN-3841.006.patch, YARN-3841.007.patch,
> YARN-3841.008.patch, YARN-3841.009.patch
>
>
> HDFS backing storage is useful for following scenarios.
> 1. For Hadoop clusters which don't run HBase.
> 2. For fallback from HBase when HBase cluster is temporary unavailable.
> Quoting ATS design document of YARN-2928:
> {quote}
> In the case the HBase
> storage is not available, the plugin should buffer the writes temporarily
> (e.g. HDFS), and flush
> them once the storage comes back online. Reading and writing to hdfs as the
> the backup storage
> could potentially use the HDFS writer plugin unless the complexity of
> generalizing the HDFS
> writer plugin for this purpose exceeds the benefits of reusing it here.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5336) Limit the flow name size & consider cleanup for hex chars
[ https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-5336: - Labels: YARN-5355 atsv2 (was: YARN-5355) > Limit the flow name size & consider cleanup for hex chars > - > > Key: YARN-5336 > URL: https://issues.apache.org/jira/browse/YARN-5336 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Sushil Ks >Priority: Major > Labels: YARN-5355, atsv2 > Fix For: 3.3.0 > > Attachments: YARN-5336.001.patch, YARN-5336.002.patch, > YARN-5336.003.patch > > > As recommended by [~jrottinghuis] , need to add in some limit (default and > configurable) for accepting key values to be written to the backend. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5336) Limit the flow name size & consider cleanup for hex chars
[ https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779848#comment-16779848 ] Vrushali C commented on YARN-5336: -- Thanks for the patch Sushil, I have committed it to trunk as part of https://github.com/apache/hadoop/commit/0ec962ac8f0fa2a7a1811efaa0258c3e2564c79a > Limit the flow name size & consider cleanup for hex chars > - > > Key: YARN-5336 > URL: https://issues.apache.org/jira/browse/YARN-5336 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Sushil Ks >Priority: Major > Labels: YARN-5355 > Attachments: YARN-5336.001.patch, YARN-5336.002.patch, > YARN-5336.003.patch > > > As recommended by [~jrottinghuis] , need to add in some limit (default and > configurable) for accepting key values to be written to the backend. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2
[
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779819#comment-16779819
]
Vrushali C commented on YARN-9150:
--
Thanks for the patch Sushil! I think the ASF license warning is unrelated to
your patch. I will commit the patch v003 shortly to branch-2
> Making TimelineSchemaCreator support different backends for Timeline Schema
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150-branch-2.001.patch,
> YARN-9150-branch-2.002.patch, YARN-9150-branch-2.003.patch,
> YARN-9150-branch-2.patch, YARN-9150.001.patch, YARN-9150.002.patch,
> jenkins_build.png
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for
> creating Timeline Schema's only for HBase, Hence creating this JIRA for
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
> Add the following property in *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.schema-creator.class
>
> YOUR_TIMELINE_SCHEMA_CREATOR_CLASS
>
> {code}
> The Command needed to run the TimelineSchemaCreator need not be changed
> i.e the below existing command can be used irrespective of the backend
> configured.
> {code:java}
> bin/hadoop
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
> -create
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3841) [Storage implementation] Adding retry semantics to HDFS backing storage
[
https://issues.apache.org/jira/browse/YARN-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778942#comment-16778942
]
Vrushali C commented on YARN-3841:
--
As discussed in the call today, I just triggered a build to ensure it is fine.
> [Storage implementation] Adding retry semantics to HDFS backing storage
> ---
>
> Key: YARN-3841
> URL: https://issues.apache.org/jira/browse/YARN-3841
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Tsuyoshi Ozawa
>Assignee: Abhishek Modi
>Priority: Major
> Labels: YARN-5355
> Attachments: YARN-3841-YARN-7055.002.patch, YARN-3841.001.patch,
> YARN-3841.002.patch, YARN-3841.003.patch, YARN-3841.004.patch,
> YARN-3841.005.patch, YARN-3841.006.patch, YARN-3841.007.patch,
> YARN-3841.008.patch, YARN-3841.009.patch
>
>
> HDFS backing storage is useful for following scenarios.
> 1. For Hadoop clusters which don't run HBase.
> 2. For fallback from HBase when HBase cluster is temporary unavailable.
> Quoting ATS design document of YARN-2928:
> {quote}
> In the case the HBase
> storage is not available, the plugin should buffer the writes temporarily
> (e.g. HDFS), and flush
> them once the storage comes back online. Reading and writing to hdfs as the
> the backup storage
> could potentially use the HDFS writer plugin unless the complexity of
> generalizing the HDFS
> writer plugin for this purpose exceeds the benefits of reusing it here.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9335) [atsv2] Restrict the number of elements held in NM timeline collector when backend is unreachable
Vrushali C created YARN-9335: Summary: [atsv2] Restrict the number of elements held in NM timeline collector when backend is unreachable Key: YARN-9335 URL: https://issues.apache.org/jira/browse/YARN-9335 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vrushali C For ATSv2 , if the backend is unreachable, the number/size of data held in timeline collector's memory increases significantly. This is not good for the NM memory. Filing jira to set a limit on how many/much should be retained by the timeline collector in memory in case the backend is not reachable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2
[
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778945#comment-16778945
]
Vrushali C commented on YARN-9150:
--
As discussed in the call today, just triggered a build to check.
> Making TimelineSchemaCreator support different backends for Timeline Schema
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150-branch-2.001.patch,
> YARN-9150-branch-2.002.patch, YARN-9150-branch-2.003.patch,
> YARN-9150-branch-2.patch, YARN-9150.001.patch, YARN-9150.002.patch,
> jenkins_build.png
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for
> creating Timeline Schema's only for HBase, Hence creating this JIRA for
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
> Add the following property in *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.schema-creator.class
>
> YOUR_TIMELINE_SCHEMA_CREATOR_CLASS
>
> {code}
> The Command needed to run the TimelineSchemaCreator need not be changed
> i.e the below existing command can be used irrespective of the backend
> configured.
> {code:java}
> bin/hadoop
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
> -create
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5336) Limit the flow name size & consider cleanup for hex chars
[ https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778941#comment-16778941 ] Vrushali C commented on YARN-5336: -- Thanks for the patch Sushil! v03 looks good to me. Committing shortly. > Limit the flow name size & consider cleanup for hex chars > - > > Key: YARN-5336 > URL: https://issues.apache.org/jira/browse/YARN-5336 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Sushil Ks >Priority: Major > Labels: YARN-5355 > Attachments: YARN-5336.001.patch, YARN-5336.002.patch, > YARN-5336.003.patch > > > As recommended by [~jrottinghuis] , need to add in some limit (default and > configurable) for accepting key values to be written to the backend. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8834) Provide Java client for fetching Yarn specific entities from TimelineReader
[ https://issues.apache.org/jira/browse/YARN-8834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-8834: - Labels: atsv2 (was: ) > Provide Java client for fetching Yarn specific entities from TimelineReader > --- > > Key: YARN-8834 > URL: https://issues.apache.org/jira/browse/YARN-8834 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Critical > Labels: atsv2 > Fix For: 3.1.2, 3.3.0, 3.2.1 > > Attachments: YARN-8834.001.patch, YARN-8834.002.patch, > YARN-8834.003.patch, YARN-8834.004.patch, YARN-8834.005.patch, > YARN-8834.006.patch > > > While reviewing YARN-8303, we felt that it is necessary to provide > TimelineReaderClient which wraps all the REST calls in it so that user can > just provide application or container ids along with filters.Currently > fetching entities from TimelineReader is only via REST call or somebody need > to write java client get entities. > It is good to provide TimelineReaderClient which fetch entities from > TimelineReaderServer. This will be more useful. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9149) yarn container -status misses logUrl when integrated with ATSv2
[
https://issues.apache.org/jira/browse/YARN-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763005#comment-16763005
]
Vrushali C commented on YARN-9149:
--
Do we need a branch-2 patch?
> yarn container -status misses logUrl when integrated with ATSv2
> ---
>
> Key: YARN-9149
> URL: https://issues.apache.org/jira/browse/YARN-9149
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9149.001.patch, YARN-9149.002.patch,
> YARN-9149.003.patch
>
>
> Post YARN-8303, yarn client can be integrated with ATSv2. But log url and
> start and end time is printing data is wrong!
> {code}
> Container Report :
> Container-Id : container_1545035586969_0001_01_01
> Start-Time : 0
> Finish-Time : 0
> State : COMPLETE
> Execution-Type : GUARANTEED
> LOG-URL : null
> Host : localhost:25006
> NodeHttpAddress : localhost:25008
> Diagnostics :
> {code}
> # TimelineEntityV2Converter#convertToContainerReport set logUrl as *null*.
> This need set for proper log url based on yarn.log.server.web-service.url
> # TimelineEntityV2Converter#convertToContainerReport parses start/end time
> wrongly. Comparison should happen with entityType but below code is doing
> entityId
> {code}
> if (events != null) {
> for (TimelineEvent event : events) {
> if (event.getId().equals(
> ContainerMetricsConstants.CREATED_IN_RM_EVENT_TYPE)) {
> createdTime = event.getTimestamp();
> } else if (event.getId().equals(
> ContainerMetricsConstants.FINISHED_IN_RM_EVENT_TYPE)) {
> finishedTime = event.getTimestamp();
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6735) Have a way to turn off container metrics from NMs
[ https://issues.apache.org/jira/browse/YARN-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762995#comment-16762995 ] Vrushali C commented on YARN-6735: -- Does this apply to branch-2 as well? > Have a way to turn off container metrics from NMs > - > > Key: YARN-6735 > URL: https://issues.apache.org/jira/browse/YARN-6735 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-6735.001.patch, YARN-6735.002.patch, > YARN-6735.003.patch > > > Have a way to turn off emitting system metrics from NMs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8219) Add application launch time to ATSV2
[ https://issues.apache.org/jira/browse/YARN-8219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762992#comment-16762992 ] Vrushali C commented on YARN-8219: -- Does this need a back-port to branch-2? I think trunk and branch are diverging a lot in terms of timeline service patches. > Add application launch time to ATSV2 > > > Key: YARN-8219 > URL: https://issues.apache.org/jira/browse/YARN-8219 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver >Reporter: Kanwaljeet Sachdev >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-8219.001.patch, YARN-8219.002.patch > > > YARN-7088 publishes application launch time to RMStore and also adds it to > the YARN UI. It would be a nice enhancement to have the launchTime event > published into the ATSV2. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8219) Add application launch time to ATSV2
[ https://issues.apache.org/jira/browse/YARN-8219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758744#comment-16758744 ] Vrushali C commented on YARN-8219: -- Thanks for the patch Abhishek! We discussed last night in the call that the launch time is the application attempt launch time and can you remind me what we said about the application launch time field if there are more than one attempts? Does this field get over-written with new values? Rohith and I were thinking this field YARN_APPLICATION_LAUNCHED should be YARN_APPLICATION_ATTEMPT_LAUNCHED in that case. > Add application launch time to ATSV2 > > > Key: YARN-8219 > URL: https://issues.apache.org/jira/browse/YARN-8219 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver >Reporter: Kanwaljeet Sachdev >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8219.001.patch, YARN-8219.002.patch > > > YARN-7088 publishes application launch time to RMStore and also adds it to > the YARN UI. It would be a nice enhancement to have the launchTime event > published into the ATSV2. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8549) Adding a NoOp timeline writer and reader plugin classes for ATSv2
[
https://issues.apache.org/jira/browse/YARN-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758740#comment-16758740
]
Vrushali C edited comment on YARN-8549 at 2/1/19 10:43 PM:
---
Hi Prabha
Can you update the patch for branch-2 as well to make it in sync with the trunk
patch? I tried applying the latest trunk v5 patch to branch-2 but it does not
compile.
{code}
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 02:35 min
[INFO] Finished at: 2019-02-01T14:34:11-08:00
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-yarn-server-timelineservice: Compilation failure: Compilation
failure:
[ERROR]
/Users/vchannapattan/Vrushali/code/Q22017_hadoop/hadoop_commit/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/NoOpTimelineWriterImpl.java:[22,58]
cannot find symbol
[ERROR] symbol: class TimelineDomain
[ERROR] location: package org.apache.hadoop.yarn.api.records.timelineservice
[ERROR]
/Users/vchannapattan/Vrushali/code/Q22017_hadoop/hadoop_commit/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/NoOpTimelineWriterImpl.java:[63,38]
cannot find symbol
[ERROR] symbol: class TimelineDomain
[ERROR] location: class
org.apache.hadoop.yarn.server.timelineservice.storage.NoOpTimelineWriterImpl
[ERROR]
/Users/vchannapattan/Vrushali/code/Q22017_hadoop/hadoop_commit/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/NoOpTimelineWriterImpl.java:[61,3]
method does not override or implement a method from a supertype
[ERROR] -> [Help 1]
[ERROR]
{code}
thanks
Vrushali
was (Author: vrushalic):
Hi Prabha
Can you update the patch for branch-2 as well to make it in sync with the trunk
patch?
thanks
Vrushali
> Adding a NoOp timeline writer and reader plugin classes for ATSv2
> -
>
> Key: YARN-8549
> URL: https://issues.apache.org/jira/browse/YARN-8549
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2, timelineclient, timelineserver
>Reporter: Prabha Manepalli
>Assignee: Prabha Manepalli
>Priority: Minor
> Attachments: YARN-8549-branch-2.03.patch,
> YARN-8549-branch-2.04.patch, YARN-8549.v1.patch, YARN-8549.v2.patch,
> YARN-8549.v4.patch, YARN-8549.v5.patch
>
>
> Stub implementation for TimeLineReader and TimeLineWriter classes.
> These are useful for functional testing of writer and reader path for ATSv2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8549) Adding a NoOp timeline writer and reader plugin classes for ATSv2
[ https://issues.apache.org/jira/browse/YARN-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758740#comment-16758740 ] Vrushali C commented on YARN-8549: -- Hi Prabha Can you update the patch for branch-2 as well to make it in sync with the trunk patch? thanks Vrushali > Adding a NoOp timeline writer and reader plugin classes for ATSv2 > - > > Key: YARN-8549 > URL: https://issues.apache.org/jira/browse/YARN-8549 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2, timelineclient, timelineserver >Reporter: Prabha Manepalli >Assignee: Prabha Manepalli >Priority: Minor > Attachments: YARN-8549-branch-2.03.patch, > YARN-8549-branch-2.04.patch, YARN-8549.v1.patch, YARN-8549.v2.patch, > YARN-8549.v4.patch, YARN-8549.v5.patch > > > Stub implementation for TimeLineReader and TimeLineWriter classes. > These are useful for functional testing of writer and reader path for ATSv2 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8549) Adding a NoOp timeline writer and reader plugin classes for ATSv2
[ https://issues.apache.org/jira/browse/YARN-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758728#comment-16758728 ] Vrushali C commented on YARN-8549: -- Committed to trunk as part of https://github.com/apache/hadoop/commit/2c135130402255ce41e1ef958989e746f21ae1ab thanks [~prabham] for the patch and thanks [~suma.shivaprasad] and [~giovanni.fumarola] for the reviews! > Adding a NoOp timeline writer and reader plugin classes for ATSv2 > - > > Key: YARN-8549 > URL: https://issues.apache.org/jira/browse/YARN-8549 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2, timelineclient, timelineserver >Reporter: Prabha Manepalli >Assignee: Prabha Manepalli >Priority: Minor > Attachments: YARN-8549-branch-2.03.patch, > YARN-8549-branch-2.04.patch, YARN-8549.v1.patch, YARN-8549.v2.patch, > YARN-8549.v4.patch, YARN-8549.v5.patch > > > Stub implementation for TimeLineReader and TimeLineWriter classes. > These are useful for functional testing of writer and reader path for ATSv2 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8549) Adding a NoOp timeline writer and reader plugin classes for ATSv2
[ https://issues.apache.org/jira/browse/YARN-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758704#comment-16758704 ] Vrushali C commented on YARN-8549: -- +1 on patch v5. I need to confirm this applies to latest trunk, although these are new classes. > Adding a NoOp timeline writer and reader plugin classes for ATSv2 > - > > Key: YARN-8549 > URL: https://issues.apache.org/jira/browse/YARN-8549 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2, timelineclient, timelineserver >Reporter: Prabha Manepalli >Assignee: Prabha Manepalli >Priority: Minor > Attachments: YARN-8549-branch-2.03.patch, > YARN-8549-branch-2.04.patch, YARN-8549.v1.patch, YARN-8549.v2.patch, > YARN-8549.v4.patch, YARN-8549.v5.patch > > > Stub implementation for TimeLineReader and TimeLineWriter classes. > These are useful for functional testing of writer and reader path for ATSv2 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3841) [Storage implementation] Adding retry semantics to HDFS backing storage
[
https://issues.apache.org/jira/browse/YARN-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757979#comment-16757979
]
Vrushali C commented on YARN-3841:
--
Thanks Abhishek for updating the patch after our last call!
I think v009 looks good. I will commit it later tonight.
> [Storage implementation] Adding retry semantics to HDFS backing storage
> ---
>
> Key: YARN-3841
> URL: https://issues.apache.org/jira/browse/YARN-3841
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Tsuyoshi Ozawa
>Assignee: Abhishek Modi
>Priority: Major
> Labels: YARN-5355
> Attachments: YARN-3841-YARN-7055.002.patch, YARN-3841.001.patch,
> YARN-3841.002.patch, YARN-3841.003.patch, YARN-3841.004.patch,
> YARN-3841.005.patch, YARN-3841.006.patch, YARN-3841.007.patch,
> YARN-3841.008.patch, YARN-3841.009.patch
>
>
> HDFS backing storage is useful for following scenarios.
> 1. For Hadoop clusters which don't run HBase.
> 2. For fallback from HBase when HBase cluster is temporary unavailable.
> Quoting ATS design document of YARN-2928:
> {quote}
> In the case the HBase
> storage is not available, the plugin should buffer the writes temporarily
> (e.g. HDFS), and flush
> them once the storage comes back online. Reading and writing to hdfs as the
> the backup storage
> could potentially use the HDFS writer plugin unless the complexity of
> generalizing the HDFS
> writer plugin for this purpose exceeds the benefits of reusing it here.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5357) Timeline service v2 integration with Federation
[
https://issues.apache.org/jira/browse/YARN-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755751#comment-16755751
]
Vrushali C commented on YARN-5357:
--
Notes from our discussion around this. Abhishek, Sushil, Prabha, Rohith and I
attended the discussion call.
Current situation:
Timeline Service collector's address & port are sent from the RM to NMs which
run containers for this app to publish the system metrics from the NM. In case
of federated yarn clusters, since an Uberized AM is used for coordinating with
containers running in foreign subclusters, the NMs in foreign sub-clusters do
not have the timeline service collector address.
This is not specific to federation but is a design item to be solved for any
unmanaged AM.
We discussed a few approaches. One of which was allowing the AM itself to send
the timeline service address & port to other foreign subclusters' RM and then
propagating it from there. But this has some issues. A malicious AM might try
to "game" the system by sending those a dummy collector address to some NMs.
This way, some NMs' system metrics mysteriously disappear before being reported
and therefore chargeback & accountability will become incorrect.
Another idea Abhishek proposed was around picking a random NM for starting up a
timeline-service collector when the AM is uberized. He will be working on the
design idea further and will discuss with his team mates and we can all review
it when he has consolidated his thoughts.
> Timeline service v2 integration with Federation
>
>
> Key: YARN-5357
> URL: https://issues.apache.org/jira/browse/YARN-5357
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
>Assignee: Prabha Manepalli
>Priority: Major
>
> Jira to note the discussion points from an initial chat about integrating
> Timeline Service v2 with Federation (YARN-2915).
> cc [~subru] [~curino]
> For Federation:
> - all entities that belong to the same flow run should have the same cluster
> name
> - app id in the same flow run strongly ordered in time
> - need a logical cluster name and physical cluster name
> - a possibility to implement the Application TimelineCollector as an
> interceptor in the AMRMProxyService.
> For Timeline Service:
> - need to store physical cluster id and logical cluster id so that we don't
> lose information at any level (flow/app/entity etc)
> - add a new table app id to cluster mapping table
> - need a different entity table/some table to store node level metrics for
> physical cluster stats. Once we get to node-level rollup, we probably have to
> store something in a dc, cluster, rack, node hierarchy. In that case a
> physical cluster makes sense, but we'd still need some way to tie physical
> and logical together in order to make automatic error detection etc that
> we're envisioning feasible within a federated setup.
> For the Cluster Naming convention:
> - three situations for cluster name:
> > app submitted to router should take federated (aka logical) cluster name
> > app submitted directly to RM should take physical cluster name
> > Info about the physical cluster in entities?
> - suggestion to set the cluster name as yarn tag at the router level (in the
> app submission context)
> Other points to note:
> - for federation to work smoothly in environments that use HDFS some
> additional considerations are needed, and possibly some solution like what is
> being used at Twitter with the nFly approach.
> Email thread context:
> {code}
> -- Forwarded message --
> From: Joep Rottinghuis
> Date: Fri, Jul 8, 2016 at 1:22 PM
> Subject: Re: Federation -Timeline Service meeting notes
> To: Subramaniam Venkatraman Krishnan
> Cc: Sangjin Lee, Vrushali Channapattan , Carlo Curino
> Thanks for the notes.
> I think that for federation to work smoothly in environments that use HDFS
> some additional considerations are needed, and possibly some solution like
> what we're using at Twitter with our nFly approach.
> bq. - need a different entity table/some table to store node level metrics
> for physical cluster stats
> Once we get to node-level rollup, we probably have to store something in a
> dc, cluster, rack, node hierarchy. In that case a physical cluster makes
> sense, but we'd still need some way to tie physical and logical together in
> order to make automatic error detection etc that we're envisioning feasible
> within a federated setup.
> Cheers,
> Joep
> On Fri, Jul 8, 2016 at 1:00 PM, Subramaniam Venkatraman Krishnan wrote:
> Thanks Vrushali for crisply capturing the essential from our rambling
> discussion J.
>
> Sangjin, I just want to add one comment to yours – we want to retain the
> physical
[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2
[
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755735#comment-16755735
]
Vrushali C commented on YARN-9150:
--
Not sure why the jenkins builds are failing, perhaps branch-2 builds are having
issues. Sushil is planning to upload a dummy patch for branch-2 with some small
change so that we can see if it is indeed a branch-2 build generic issue.
> Making TimelineSchemaCreator support different backends for Timeline Schema
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150-branch-2.patch, YARN-9150.001.patch,
> YARN-9150.002.patch, jenkins_build.png
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for
> creating Timeline Schema's only for HBase, Hence creating this JIRA for
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
> Add the following property in *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.schema-creator.class
>
> YOUR_TIMELINE_SCHEMA_CREATOR_CLASS
>
> {code}
> The Command needed to run the TimelineSchemaCreator need not be changed
> i.e the below existing command can be used irrespective of the backend
> configured.
> {code:java}
> bin/hadoop
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
> -create
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3488) AM get timeline service info from RM rather than Application specific configuration.
[ https://issues.apache.org/jira/browse/YARN-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755736#comment-16755736 ] Vrushali C commented on YARN-3488: -- Thanks Abhishek! As per our chat in the call today, we can have a separate jira for the mapreduce AM changes. For the distributed shell we can include the changes in this jira itself. > AM get timeline service info from RM rather than Application specific > configuration. > > > Key: YARN-3488 > URL: https://issues.apache.org/jira/browse/YARN-3488 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications >Reporter: Junping Du >Assignee: Abhishek Modi >Priority: Major > Labels: YARN-5355 > Attachments: YARN-3488.001.patch, YARN-3488.002.patch > > > Since v1 timeline service, we have MR configuration to enable/disable putting > history event to timeline service. For today's v2 timeline service ongoing > effort, currently we have different methods/structures between v1 and v2 for > consuming TimelineClient, so application have to be aware of which version > timeline service get used there. > There are basically two options here: > First option is as current way in DistributedShell or MR to let application > has specific configuration to point out that if enabling ATS and which > version could be, like: MRJobConfig.MAPREDUCE_JOB_EMIT_TIMELINE_DATA, etc. > The other option is to let application to figure out timeline related info > from YARN/RM, it can be done through registerApplicationMaster() in > ApplicationMasterProtocol with return value for service "off", "v1_on", or > "v2_on". > We prefer the latter option because application owner doesn't have to aware > RM/YARN infrastructure details. Please note that we should keep compatible > (consistent behavior with the same setting) with released configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7271) Add a yarn application cost calculation framework in TimelineService v2
[ https://issues.apache.org/jira/browse/YARN-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755305#comment-16755305 ] Vrushali C commented on YARN-7271: -- Thanks for picking this up Sushil! Let's chat about this in our call sometime when you have had time to look through Eric's jira and have consolidated your thoughts around this. I am very interested and invested in fixing this jira. > Add a yarn application cost calculation framework in TimelineService v2 > --- > > Key: YARN-7271 > URL: https://issues.apache.org/jira/browse/YARN-7271 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Sushil Ks >Priority: Major > > Timeline Service v2 captures information about a yarn application. From this > info, we would like to calculate the "cost" of an yarn application. This > would be rolled up to the flow level as well (and user and queue level > eventually). > We need a way to accept machine cost (TCO per day) and enable this > calculation. This will help in chargeback for yarn apps. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5336) Limit the flow name size & consider cleanup for hex chars
[ https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751899#comment-16751899 ] Vrushali C commented on YARN-5336: -- Also, if we are updating the patch, could you add some comments around the config variables added in YarnConfiguration.java? > Limit the flow name size & consider cleanup for hex chars > - > > Key: YARN-5336 > URL: https://issues.apache.org/jira/browse/YARN-5336 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Sushil Ks >Priority: Major > Labels: YARN-5355 > Attachments: YARN-5336.001.patch, YARN-5336.002.patch > > > As recommended by [~jrottinghuis] , need to add in some limit (default and > configurable) for accepting key values to be written to the backend. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5336) Limit the flow name size & consider cleanup for hex chars
[
https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751898#comment-16751898
]
Vrushali C commented on YARN-5336:
--
Thanks Sushil! Patch 002 looks better but there is one place L206 in
TimelineUtils.java.
I was wondering about the following, what do you think:
{code}
if (length <= 0) {
length = flowName.length();
}
return flowName.substring(0, length);
{code}
>From Java 1.7.0_06, that String.substring now has a linear complexity instead
>of a constant one. Reference
>bug:https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4513622
So:
- let's update code to use StringUtils for the substring.
- Also, when length is <=0, let's simply return the flowname instead of
calculating the entire substring again as the flowname. This will help improve
the time performance of the code.
> Limit the flow name size & consider cleanup for hex chars
> -
>
> Key: YARN-5336
> URL: https://issues.apache.org/jira/browse/YARN-5336
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
>Assignee: Sushil Ks
>Priority: Major
> Labels: YARN-5355
> Attachments: YARN-5336.001.patch, YARN-5336.002.patch
>
>
> As recommended by [~jrottinghuis] , need to add in some limit (default and
> configurable) for accepting key values to be written to the backend.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2
[
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743649#comment-16743649
]
Vrushali C commented on YARN-9150:
--
The retriggered Jenkins Build
https://builds.apache.org/job/Hadoop-trunk-Commit/15779/
has succeeded. It did not update the jira so attached a screenshot of the
successful build to the jira.
> Making TimelineSchemaCreator support different backends for Timeline Schema
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch, YARN-9150.002.patch,
> jenkins_build.png
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for
> creating Timeline Schema's only for HBase, Hence creating this JIRA for
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
> Add the following property in *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.schema-creator.class
> YOUR_TIMELINE_SCHEMA_CREATOR_CLASS
>
> {code}
> The Command needed to run the TimelineSchemaCreator need not be changed
> i.e the below existing command can be used irrespective of the backend
> configured.
> {code:java}
> bin/hadoop
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
> -create
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2
[
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-9150:
-
Attachment: jenkins_build.png
> Making TimelineSchemaCreator support different backends for Timeline Schema
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch, YARN-9150.002.patch,
> jenkins_build.png
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for
> creating Timeline Schema's only for HBase, Hence creating this JIRA for
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
> Add the following property in *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.schema-creator.class
> YOUR_TIMELINE_SCHEMA_CREATOR_CLASS
>
> {code}
> The Command needed to run the TimelineSchemaCreator need not be changed
> i.e the below existing command can be used irrespective of the backend
> configured.
> {code:java}
> bin/hadoop
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
> -create
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2
[
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743642#comment-16743642
]
Vrushali C commented on YARN-9150:
--
Retriggered jenkins build
https://builds.apache.org/job/Hadoop-trunk-Commit/15779/
> Making TimelineSchemaCreator support different backends for Timeline Schema
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch, YARN-9150.002.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for
> creating Timeline Schema's only for HBase, Hence creating this JIRA for
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
> Add the following property in *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.schema-creator.class
> YOUR_TIMELINE_SCHEMA_CREATOR_CLASS
>
> {code}
> The Command needed to run the TimelineSchemaCreator need not be changed
> i.e the below existing command can be used irrespective of the backend
> configured.
> {code:java}
> bin/hadoop
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
> -create
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2
[
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743630#comment-16743630
]
Vrushali C edited comment on YARN-9150 at 1/16/19 5:28 AM:
---
Committed to trunk as part of
https://github.com/apache/hadoop/commit/6be39230a67098e7d157925575b3b18bbf947717
and
https://github.com/apache/hadoop/commit/713ded6b15dc0b5e4205a7812a62225377e0b32b
was (Author: vrushalic):
Committed to trunk as part of
https://github.com/apache/hadoop/commit/6be39230a67098e7d157925575b3b18bbf947717
> Making TimelineSchemaCreator support different backends for Timeline Schema
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch, YARN-9150.002.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for
> creating Timeline Schema's only for HBase, Hence creating this JIRA for
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
> Add the following property in *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.schema-creator.class
> YOUR_TIMELINE_SCHEMA_CREATOR_CLASS
>
> {code}
> The Command needed to run the TimelineSchemaCreator need not be changed
> i.e the below existing command can be used irrespective of the backend
> configured.
> {code:java}
> bin/hadoop
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
> -create
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2
[
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743630#comment-16743630
]
Vrushali C commented on YARN-9150:
--
Committed to trunk as part of
https://github.com/apache/hadoop/commit/6be39230a67098e7d157925575b3b18bbf947717
> Making TimelineSchemaCreator support different backends for Timeline Schema
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch, YARN-9150.002.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for
> creating Timeline Schema's only for HBase, Hence creating this JIRA for
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
> Add the following property in *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.schema-creator.class
> YOUR_TIMELINE_SCHEMA_CREATOR_CLASS
>
> {code}
> The Command needed to run the TimelineSchemaCreator need not be changed
> i.e the below existing command can be used irrespective of the backend
> configured.
> {code:java}
> bin/hadoop
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
> -create
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2
[
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743620#comment-16743620
]
Vrushali C commented on YARN-9150:
--
Committing this now.
[~Sushil-K-S] could you also upload a patch for branch-2? Same one should work
but please confirm and upload one.
> Making TimelineSchemaCreator support different backends for Timeline Schema
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch, YARN-9150.002.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for
> creating Timeline Schema's only for HBase, Hence creating this JIRA for
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
> Add the following property in *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.schema-creator.class
> YOUR_TIMELINE_SCHEMA_CREATOR_CLASS
>
> {code}
> The Command needed to run the TimelineSchemaCreator need not be changed
> i.e the below existing command can be used irrespective of the backend
> configured.
> {code:java}
> bin/hadoop
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
> -create
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2
[
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742413#comment-16742413
]
Vrushali C commented on YARN-9150:
--
thanks [~Sushil-K-S] !
+ 1 on patch 002.
One minor comment but we can update it later. Usually the default config
variable names have the "DEFAULT" at the end or somewhere in the middle of the
word. The reason is that, when someone is coding in an IDE, the dropdown would
show the possible matches so both the config variable name and the default
variable name show up as we start typing TIMELINE_SERVICE_SCHEMA_CREATOR... So
we can have TIMELINE_SERVICE_SCHEMA_CREATOR_CLASS and
TIMELINE_SERVICE_SCHEMA_CREATOR_DEFAULT_CLASS
But I will commit this patch (in a day). We can update this later on. We also
need to add to the documentation, we can do it as part of the YARN-9016 patch.
> Making TimelineSchemaCreator support different backends for Timeline Schema
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch, YARN-9150.002.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for
> creating Timeline Schema's only for HBase, Hence creating this JIRA for
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
> Add the following property in *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.schema-creator.class
> YOUR_TIMELINE_SCHEMA_CREATOR_CLASS
>
> {code}
> The Command needed to run the TimelineSchemaCreator need not be changed
> i.e the below existing command can be used irrespective of the backend
> configured.
> {code:java}
> bin/hadoop
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
> -create
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5336) Limit the flow name size & consider cleanup for hex chars
[ https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16741786#comment-16741786 ] Vrushali C commented on YARN-5336: -- Thanks for the patch Sushil! I think I might file another jira to track limiting the size of the key/value for ALL key-value pairs being stored. The patch looks good overall, just a few minor comments: - I think we should rename FLOW_NAME_SIZE to FLOW_NAME_MAX_SIZE - We can also add in a FLOW_NAME_DEFAULT_MAX_SIZE which is set to 0. And we document that 0 indicates no restriction on the size. - As Abhishek suggested, we can also add these to the yarn site default xml. - Let's use Apache Commons Lang’s StringUtils.replace instead of the String.replaceAll at L209 in TimelineUtils.java > Limit the flow name size & consider cleanup for hex chars > - > > Key: YARN-5336 > URL: https://issues.apache.org/jira/browse/YARN-5336 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Sushil Ks >Priority: Major > Labels: YARN-5355 > Attachments: YARN-5336.001.patch > > > As recommended by [~jrottinghuis] , need to add in some limit (default and > configurable) for accepting key values to be written to the backend. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2
[
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737813#comment-16737813
]
Vrushali C commented on YARN-9150:
--
Hi Sushil
The patch looks fine to me. I am trying to find the old jenkins build it is no
longer present. Let me try resubmitting the patch to see what jenkins says.
> Making TimelineSchemaCreator support different backends for Timeline Schema
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for
> creating Timeline Schema's only for HBase, Hence creating this JIRA for
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
> Add the following property in *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.schema-creator.class
> YOUR_TIMELINE_SCHEMA_CREATOR_CLASS
>
> {code}
> The Command needed to run the TimelineSchemaCreator need not be changed
> i.e the below existing command can be used irrespective of the backend
> configured.
> {code:java}
> bin/hadoop
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
> -create
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2
[
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720973#comment-16720973
]
Vrushali C commented on YARN-9016:
--
Also, we have been looking at the licenses of modules & their transitive
dependencies used in this patch, it appears that all licenses are fine to be
included (some are already being used in the Apache Hadoop codebase like mit,
json etc). Sushil will add more details.
> DocumentStore as a backend for ATSv2
>
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch
>
>
> h1. Document Store for ATSv2
> The Document Store for ATSv2 is a framework for plugging in
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB ,
> MongoDB, ElasticSearch etc.
> * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch,
> MongoDB etc by just adding new configurations properties and writing Document
> Store reader and writer clients.
> * Currently has support for CosmosDB.
> * All writes are Async and buffered, latest document would be flushed to the
> store either if the document buffer gets full or periodically at every flush
> interval in background without adding any additional latency to the running
> jobs..
> * All the REST API's of Timeline Reader Server are supported.
> h4.
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.writer.class/name>
>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
>
>
>yarn.timeline-service.reader.class/name>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
>
>
>yarn.timeline-service.documentstore.db-name
>YOUR_DATABASE_NAME
> {code}
> h3. *Creating DB and Collections for storing documents*
> This is similar to HBase *TimelineSchemaCreator* the
> following command needs to be executed once for setting up the database and
> collections for storing documents.
> {code:java}
> hadoop
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3. *Azure CosmosDB*
> To use Azure CosmosDB as a DocumentStore for ATSv2, the additional
> properties under *yarn-site.xml* is required..
> {code:java}
>
>
>yarn.timeline-service.store-type
>COSMOS_DB
>
>
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
>
>
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
>
> {code}
>
> *Testing locally*
> In order to test the Azure CosmosDB as a DocumentStore
> locally, install the emulator from
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and
> start it locally. Set the endpoint and master key under *yarn-site.xml* as
> mentioned above and run any example job like DistributedShell etc. Later you
> can check the data explorer UI of Azure CosmosDB locally to query the
> documents or even launch the *TimelineReader* locally and fetch/query the
> data from REST API's.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2
[
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720972#comment-16720972
]
Vrushali C commented on YARN-9016:
--
As discussed in the call just now, Sushil will file another jira to refactor
the schema creator/ hbase code and then rebase this cosmos db patch.
> DocumentStore as a backend for ATSv2
>
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch
>
>
> h1. Document Store for ATSv2
> The Document Store for ATSv2 is a framework for plugging in
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB ,
> MongoDB, ElasticSearch etc.
> * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch,
> MongoDB etc by just adding new configurations properties and writing Document
> Store reader and writer clients.
> * Currently has support for CosmosDB.
> * All writes are Async and buffered, latest document would be flushed to the
> store either if the document buffer gets full or periodically at every flush
> interval in background without adding any additional latency to the running
> jobs..
> * All the REST API's of Timeline Reader Server are supported.
> h4.
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.writer.class/name>
>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
>
>
>yarn.timeline-service.reader.class/name>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
>
>
>yarn.timeline-service.documentstore.db-name
>YOUR_DATABASE_NAME
> {code}
> h3. *Creating DB and Collections for storing documents*
> This is similar to HBase *TimelineSchemaCreator* the
> following command needs to be executed once for setting up the database and
> collections for storing documents.
> {code:java}
> hadoop
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3. *Azure CosmosDB*
> To use Azure CosmosDB as a DocumentStore for ATSv2, the additional
> properties under *yarn-site.xml* is required..
> {code:java}
>
>
>yarn.timeline-service.store-type
>COSMOS_DB
>
>
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
>
>
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
>
> {code}
>
> *Testing locally*
> In order to test the Azure CosmosDB as a DocumentStore
> locally, install the emulator from
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and
> start it locally. Set the endpoint and master key under *yarn-site.xml* as
> mentioned above and run any example job like DistributedShell etc. Later you
> can check the data explorer UI of Azure CosmosDB locally to query the
> documents or even launch the *TimelineReader* locally and fetch/query the
> data from REST API's.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5336) Limit the flow name size & consider cleanup for hex chars
[ https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704308#comment-16704308 ] Vrushali C commented on YARN-5336: -- We can consider limiting the flow name size as a start. Consider accepting a size limit. A size of 0 means no restriction and otherwise, we can cleanup the value and then limit the size. > Limit the flow name size & consider cleanup for hex chars > - > > Key: YARN-5336 > URL: https://issues.apache.org/jira/browse/YARN-5336 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Sushil Ks >Priority: Major > Labels: YARN-5355 > > As recommended by [~jrottinghuis] , need to add in some limit (default and > configurable) for accepting key values to be written to the backend. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5336) Limit the flow name size & consider cleanup for hex chars
[ https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-5336: - Description: As recommended by [~jrottinghuis] , need to add in some limit (default and configurable) for accepting key values to be written to the backend. was: As recommended by [~jrottinghuis] , need to add in some limit (default and configurable) for accepting key values to be written to the backend. > Limit the flow name size & consider cleanup for hex chars > - > > Key: YARN-5336 > URL: https://issues.apache.org/jira/browse/YARN-5336 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Sushil Ks >Priority: Major > Labels: YARN-5355 > > As recommended by [~jrottinghuis] , need to add in some limit (default and > configurable) for accepting key values to be written to the backend. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5336) Limit the flow name size & consider cleanup for hex chars
[ https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-5336: - Summary: Limit the flow name size & consider cleanup for hex chars (was: Limit the flow name size & consider cleanup for hex chars etc) > Limit the flow name size & consider cleanup for hex chars > - > > Key: YARN-5336 > URL: https://issues.apache.org/jira/browse/YARN-5336 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Sushil Ks >Priority: Major > Labels: YARN-5355 > > As recommended by [~jrottinghuis] , need to add in some limit (default and > configurable) for accepting key values to be written to the backend. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5336) Limit the flow name size & consider cleanup for hex chars etc
[ https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-5336: - Summary: Limit the flow name size & consider cleanup for hex chars etc (was: Put in some limit for accepting key-values in hbase writer) > Limit the flow name size & consider cleanup for hex chars etc > - > > Key: YARN-5336 > URL: https://issues.apache.org/jira/browse/YARN-5336 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Sushil Ks >Priority: Major > Labels: YARN-5355 > > As recommended by [~jrottinghuis] , need to add in some limit (default and > configurable) for accepting key values to be written to the backend. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5336) Put in some limit for accepting key-values in hbase writer
[ https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C reassigned YARN-5336: Assignee: Sushil Ks (was: Vrushali C) > Put in some limit for accepting key-values in hbase writer > -- > > Key: YARN-5336 > URL: https://issues.apache.org/jira/browse/YARN-5336 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Sushil Ks >Priority: Major > Labels: YARN-5355 > > As recommended by [~jrottinghuis] , need to add in some limit (default and > configurable) for accepting key values to be written to the backend. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6931) Make the aggregation interval in AppLevelTimelineCollector configurable
[ https://issues.apache.org/jira/browse/YARN-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692476#comment-16692476 ] Vrushali C commented on YARN-6931: -- Hi everyone I think this can go onto branch-2 as well, right? It does not break any compatibility or existing behavior. Other than the fact that the patch may or may not apply directly to the codebase now, is there any reason for this not to go into branch-2? cc [~abmodi] [~rohithsharma] ? > Make the aggregation interval in AppLevelTimelineCollector configurable > --- > > Key: YARN-6931 > URL: https://issues.apache.org/jira/browse/YARN-6931 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Abhishek Modi >Priority: Minor > Fix For: 3.2.0 > > Attachments: YARN-6931.001.patch, YARN-6931.002.patch > > > We do application-level metrics aggregation in AppLevelTimelineCollector, but > the interval is hardcoded. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8936) Update ATSv2 hbase.two.version to 2.0.2
[ https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-8936: - Attachment: YARN-8936-yarn-7055.0001.patch > Update ATSv2 hbase.two.version to 2.0.2 > --- > > Key: YARN-8936 > URL: https://issues.apache.org/jira/browse/YARN-8936 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2, 3.3.0 >Reporter: Rohith Sharma K S >Assignee: Vrushali C >Priority: Major > Attachments: YARN-8936-yarn-7055.0001.patch, YARN-8936.0001.patch, > YARN-8936.0002.patch, YARN-8936.0003.patch > > > Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable > Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches. > cc:/ [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8936) Update ATSv2 hbase.two.version to 2.0.2
[ https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-8936: - Attachment: YARN-8936.0003.patch > Update ATSv2 hbase.two.version to 2.0.2 > --- > > Key: YARN-8936 > URL: https://issues.apache.org/jira/browse/YARN-8936 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2, 3.3.0 >Reporter: Rohith Sharma K S >Assignee: Vrushali C >Priority: Major > Attachments: YARN-8936.0001.patch, YARN-8936.0002.patch, > YARN-8936.0003.patch > > > Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable > Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches. > cc:/ [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8107) Give an informative message when incorrect format is used in ATSv2 filter attributes
[
https://issues.apache.org/jira/browse/YARN-8107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-8107:
-
Fix Version/s: (was: 2.9.2)
> Give an informative message when incorrect format is used in ATSv2 filter
> attributes
>
>
> Key: YARN-8107
> URL: https://issues.apache.org/jira/browse/YARN-8107
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Reporter: Charan Hebri
>Assignee: Rohith Sharma K S
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 3.1.1, 3.0.3
>
> Attachments: YARN-8107.001.patch, YARN-8107.002.patch
>
>
> Using an incorrect format for infofilters, conffilters and metricfilters
> throws a NPE with no clear message to the caller. This should be tagged as a
> 400 Bad Request with an informative message. Below is the timeline reader log.
> {noformat}
> 2018-04-02 06:44:10,451 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:handleException(173)) - Processed URL
> /ws/v2/timeline/users/hrt_qa/flows/flow4/runs/1/apps?infofilters=UIDeq but
> encountered exception (Took 0 ms.)
> 2018-04-02 06:44:10,451 ERROR reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:handleException(188)) - Error while
> processing REST request
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.filter.TimelineFilterUtils.createHBaseFilterList(TimelineFilterUtils.java:276)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.reader.ApplicationEntityReader.constructFilterListBasedOnFilters(ApplicationEntityReader.java:126)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.reader.TimelineEntityReader.createFilterList(TimelineEntityReader.java:157)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.reader.TimelineEntityReader.readEntities(TimelineEntityReader.java:277)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl.getEntities(HBaseTimelineReaderImpl.java:87)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderManager.getEntities(TimelineReaderManager.java:143)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:605)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getFlowRunApps(TimelineReaderWebServices.java:1962)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
> at
>
[jira] [Updated] (YARN-8107) Give an informative message when incorrect format is used in ATSv2 filter attributes
[
https://issues.apache.org/jira/browse/YARN-8107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-8107:
-
Fix Version/s: 2.9.2
> Give an informative message when incorrect format is used in ATSv2 filter
> attributes
>
>
> Key: YARN-8107
> URL: https://issues.apache.org/jira/browse/YARN-8107
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Reporter: Charan Hebri
>Assignee: Rohith Sharma K S
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 3.1.1, 3.0.3
>
> Attachments: YARN-8107.001.patch, YARN-8107.002.patch
>
>
> Using an incorrect format for infofilters, conffilters and metricfilters
> throws a NPE with no clear message to the caller. This should be tagged as a
> 400 Bad Request with an informative message. Below is the timeline reader log.
> {noformat}
> 2018-04-02 06:44:10,451 INFO reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:handleException(173)) - Processed URL
> /ws/v2/timeline/users/hrt_qa/flows/flow4/runs/1/apps?infofilters=UIDeq but
> encountered exception (Took 0 ms.)
> 2018-04-02 06:44:10,451 ERROR reader.TimelineReaderWebServices
> (TimelineReaderWebServices.java:handleException(188)) - Error while
> processing REST request
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.filter.TimelineFilterUtils.createHBaseFilterList(TimelineFilterUtils.java:276)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.reader.ApplicationEntityReader.constructFilterListBasedOnFilters(ApplicationEntityReader.java:126)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.reader.TimelineEntityReader.createFilterList(TimelineEntityReader.java:157)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.reader.TimelineEntityReader.readEntities(TimelineEntityReader.java:277)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl.getEntities(HBaseTimelineReaderImpl.java:87)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderManager.getEntities(TimelineReaderManager.java:143)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:605)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getFlowRunApps(TimelineReaderWebServices.java:1962)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
> at
>
[jira] [Commented] (YARN-8856) TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError
[ https://issues.apache.org/jira/browse/YARN-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688445#comment-16688445 ] Vrushali C commented on YARN-8856: -- Thanks [~goiri] for the commit. > TestTimelineReaderWebServicesHBaseStorage tests failing with > NoClassDefFoundError > - > > Key: YARN-8856 > URL: https://issues.apache.org/jira/browse/YARN-8856 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jason Lowe >Assignee: Sushil Ks >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-8856.001.patch > > > TestTimelineReaderWebServicesHBaseStorage has been failing in nightly builds > with NoClassDefFoundError in the tests. Sample error and stacktrace to > follow. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8856) TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError
[ https://issues.apache.org/jira/browse/YARN-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688445#comment-16688445 ] Vrushali C edited comment on YARN-8856 at 11/15/18 5:59 PM: Thanks [~elgoiri] for the commit. was (Author: vrushalic): Thanks [~goiri] for the commit. > TestTimelineReaderWebServicesHBaseStorage tests failing with > NoClassDefFoundError > - > > Key: YARN-8856 > URL: https://issues.apache.org/jira/browse/YARN-8856 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jason Lowe >Assignee: Sushil Ks >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-8856.001.patch > > > TestTimelineReaderWebServicesHBaseStorage has been failing in nightly builds > with NoClassDefFoundError in the tests. Sample error and stacktrace to > follow. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8856) TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError
[ https://issues.apache.org/jira/browse/YARN-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682060#comment-16682060 ] Vrushali C commented on YARN-8856: -- Patch LGTM. I am downloading it and checking it locally once. > TestTimelineReaderWebServicesHBaseStorage tests failing with > NoClassDefFoundError > - > > Key: YARN-8856 > URL: https://issues.apache.org/jira/browse/YARN-8856 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jason Lowe >Assignee: Sushil Ks >Priority: Major > Attachments: YARN-8856.001.patch > > > TestTimelineReaderWebServicesHBaseStorage has been failing in nightly builds > with NoClassDefFoundError in the tests. Sample error and stacktrace to > follow. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8936) Update ATSv2 hbase.two.version to 2.0.2
[ https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1666#comment-1666 ] Vrushali C commented on YARN-8936: -- Added patch for updating hbase server 1 to 1.4.8 as well > Update ATSv2 hbase.two.version to 2.0.2 > --- > > Key: YARN-8936 > URL: https://issues.apache.org/jira/browse/YARN-8936 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2, 3.3.0 >Reporter: Rohith Sharma K S >Assignee: Vrushali C >Priority: Major > Attachments: YARN-8936.0001.patch, YARN-8936.0002.patch > > > Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable > Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches. > cc:/ [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8936) Update ATSv2 hbase.two.version to 2.0.2
[ https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-8936: - Attachment: YARN-8936.0002.patch > Update ATSv2 hbase.two.version to 2.0.2 > --- > > Key: YARN-8936 > URL: https://issues.apache.org/jira/browse/YARN-8936 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2, 3.3.0 >Reporter: Rohith Sharma K S >Assignee: Vrushali C >Priority: Major > Attachments: YARN-8936.0001.patch, YARN-8936.0002.patch > > > Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable > Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches. > cc:/ [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8936) Update ATSv2 hbase.two.version to 2.0.2
[ https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662791#comment-16662791 ] Vrushali C commented on YARN-8936: -- [~rohithsharma] do you want to try the build once for hbase 2 with this patch? I am unable to figure out how to get jenkins to build for this profile. > Update ATSv2 hbase.two.version to 2.0.2 > --- > > Key: YARN-8936 > URL: https://issues.apache.org/jira/browse/YARN-8936 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2, 3.3.0 >Reporter: Rohith Sharma K S >Assignee: Vrushali C >Priority: Major > Attachments: YARN-8936.0001.patch > > > Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable > Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches. > cc:/ [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8936) Update ATSv2 hbase.two.version to 2.0.2
[
https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662788#comment-16662788
]
Vrushali C commented on YARN-8936:
--
So the compilation and unit tests pass with -Dhbase.profile=2.0 .
The only failures are for TestTimelineReaderWebServicesHBaseStorage.
{code}
[ERROR] Failures:
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetAppNotPresent:->AbstractTimelineReaderHBaseTestBase.verifyHttpResponse:140
Response from server should have been Not Found
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetFlowRunNotPresent:2192->AbstractTimelineReaderHBaseTestBase.verifyHttpResponse:140
Response from server should have been Not Found
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testUIDNotProperlyEscaped:905->AbstractTimelineReaderHBaseTestBase.verifyHttpResponse:140
Response from server should have been Bad Request
[ERROR] Errors:
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testForFlowAppsPagination:2375->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testForFlowRunAppsPagination:2420->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testForFlowRunsPagination:2465->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGenericEntitiesForPagination:2272->verifyEntitiesForPagination:2288->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetApp:1024->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetAppWithoutFlowInfo:1064->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetAppsMetricsRange:2516->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesByUID:662->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesConfigFilters:1263->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesDataToRetrieve:1154->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesEventFilters:1640->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesInfoFilters:1380->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricFilters:1494->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesMetricsTimeRange:1820->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesRelationFilters:1696->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetEntitiesWithoutFlowInfo:1130->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetEntityDataToRetrieve:1905->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetEntityWithoutFlowInfo:1113->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetFlowApps:2047->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetFlowAppsFilters:2153->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetFlowAppsNotPresent:2253->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetFlowRun:443->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetFlowRunApps:1984->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetFlowRunAppsNotPresent:2235->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetFlowRuns:488->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetFlowRunsMetricsToRetrieve:616->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
TestTimelineReaderWebServicesHBaseStorage.testGetFlows:918->verifyFlowEntites:2349->AbstractTimelineReaderHBaseTestBase.getResponse:129
» IO
[ERROR]
[jira] [Updated] (YARN-8936) Update ATSv2 hbase.two.version to 2.0.2
[ https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-8936: - Attachment: YARN-8936.0001.patch > Update ATSv2 hbase.two.version to 2.0.2 > --- > > Key: YARN-8936 > URL: https://issues.apache.org/jira/browse/YARN-8936 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2, 3.3.0 >Reporter: Rohith Sharma K S >Priority: Major > Attachments: YARN-8936.0001.patch > > > Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable > Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches. > cc:/ [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8936) Update ATSv2 hbase.two.version to 2.0.2
[ https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C reassigned YARN-8936: Assignee: Vrushali C > Update ATSv2 hbase.two.version to 2.0.2 > --- > > Key: YARN-8936 > URL: https://issues.apache.org/jira/browse/YARN-8936 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2, 3.3.0 >Reporter: Rohith Sharma K S >Assignee: Vrushali C >Priority: Major > Attachments: YARN-8936.0001.patch > > > Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable > Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches. > cc:/ [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5742) Serve aggregated logs of historical apps from timeline service
[
https://issues.apache.org/jira/browse/YARN-5742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647455#comment-16647455
]
Vrushali C commented on YARN-5742:
--
Okay sounds good .. I tried with the same patch for branch-2 but the patch does
not apply as is to branch-2. Getting the following compilation errors.
{code}
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 02:33 min
[INFO] Finished at: 2018-10-11T21:28:15-07:00
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-yarn-server-common: Compilation failure: Compilation failure:
[ERROR]
hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogWebService.java:[34,30]
cannot find symbol
[ERROR] symbol: class JettyUtils
[ERROR] location: package org.apache.hadoop.http
[ERROR]
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogWebService.java:[338,45]
cannot find symbol
[ERROR] symbol: variable JettyUtils
[ERROR] location: class org.apache.hadoop.yarn.server.webapp.LogWebService
[ERROR]
hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogWebService.java:[473,16]
cannot find symbol
[ERROR] symbol: method getStatusInfo()
[ERROR] location: variable resp of type
com.sun.jersey.api.client.ClientResponse
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-yarn-server-common
{code}
> Serve aggregated logs of historical apps from timeline service
> --
>
> Key: YARN-5742
> URL: https://issues.apache.org/jira/browse/YARN-5742
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Rohith Sharma K S
>Priority: Critical
> Fix For: 3.2.0, 3.0.3, 3.1.2, 3.3.0
>
> Attachments: YARN-5742-POC-v0.patch, YARN-5742.01.patch,
> YARN-5742.02.patch, YARN-5742.03.patch, YARN-5742.04.patch, YARN-5742.v0.patch
>
>
> ATSv1.5 daemon has servlet to serve aggregated logs. But enabling only ATSv2,
> does not serve logs from CLI and UI for completed application. Log serving
> story has completely broken in ATSv2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3879) [Storage implementation] Create HDFS backing storage implementation for ATS reads
[ https://issues.apache.org/jira/browse/YARN-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647425#comment-16647425 ] Vrushali C commented on YARN-3879: -- Committed to branch-2 as part of https://github.com/apache/hadoop/commit/7ed627af6b3503e2b5446e582c83678218996d72 > [Storage implementation] Create HDFS backing storage implementation for ATS > reads > - > > Key: YARN-3879 > URL: https://issues.apache.org/jira/browse/YARN-3879 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Tsuyoshi Ozawa >Assignee: Abhishek Modi >Priority: Major > Labels: YARN-5355, YARN-7055 > Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.3, 3.1.2 > > Attachments: YARN-3879-YARN-7055.001.patch, YARN-3879.001.patch, > YARN-3879.002.patch, YARN-3879.003.patch, YARN-3879.004.patch, > YARN-3879.005.patch, YARN-3879.006.patch > > > Reader version of YARN-3841 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3879) [Storage implementation] Create HDFS backing storage implementation for ATS reads
[ https://issues.apache.org/jira/browse/YARN-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-3879: - Fix Version/s: 2.9.2 2.10.0 > [Storage implementation] Create HDFS backing storage implementation for ATS > reads > - > > Key: YARN-3879 > URL: https://issues.apache.org/jira/browse/YARN-3879 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Tsuyoshi Ozawa >Assignee: Abhishek Modi >Priority: Major > Labels: YARN-5355, YARN-7055 > Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.3, 3.1.2 > > Attachments: YARN-3879-YARN-7055.001.patch, YARN-3879.001.patch, > YARN-3879.002.patch, YARN-3879.003.patch, YARN-3879.004.patch, > YARN-3879.005.patch, YARN-3879.006.patch > > > Reader version of YARN-3841 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3879) [Storage implementation] Create HDFS backing storage implementation for ATS reads
[ https://issues.apache.org/jira/browse/YARN-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-3879: - Labels: YARN-5355 YARN-7055 (was: YARN-5355) > [Storage implementation] Create HDFS backing storage implementation for ATS > reads > - > > Key: YARN-3879 > URL: https://issues.apache.org/jira/browse/YARN-3879 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Tsuyoshi Ozawa >Assignee: Abhishek Modi >Priority: Major > Labels: YARN-5355, YARN-7055 > Fix For: 3.2.0, 3.0.3, 3.1.2 > > Attachments: YARN-3879-YARN-7055.001.patch, YARN-3879.001.patch, > YARN-3879.002.patch, YARN-3879.003.patch, YARN-3879.004.patch, > YARN-3879.005.patch, YARN-3879.006.patch > > > Reader version of YARN-3841 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8834) Provide Java client for fetching Yarn specific entities from TimelineReader
[ https://issues.apache.org/jira/browse/YARN-8834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647369#comment-16647369 ] Vrushali C edited comment on YARN-8834 at 10/12/18 4:17 AM: Committed to trunk as part of https://github.com/apache/hadoop/commit/a3edfddcf7822ea13bdf4858672eb82cea5e0b5f Thanks [~abmodi] for the patch and [~rohithsharma] for the reviews! Do we need some documentation examples for this? was (Author: vrushalic): Committed to trunk as part of https://github.com/apache/hadoop/commit/a3edfddcf7822ea13bdf4858672eb82cea5e0b5f Do we need some documentation examples for this? > Provide Java client for fetching Yarn specific entities from TimelineReader > --- > > Key: YARN-8834 > URL: https://issues.apache.org/jira/browse/YARN-8834 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Critical > Fix For: 3.2.0, 3.0.3, 3.1.2, 3.3.0 > > Attachments: YARN-8834.001.patch, YARN-8834.002.patch, > YARN-8834.003.patch, YARN-8834.004.patch, YARN-8834.005.patch, > YARN-8834.006.patch > > > While reviewing YARN-8303, we felt that it is necessary to provide > TimelineReaderClient which wraps all the REST calls in it so that user can > just provide application or container ids along with filters.Currently > fetching entities from TimelineReader is only via REST call or somebody need > to write java client get entities. > It is good to provide TimelineReaderClient which fetch entities from > TimelineReaderServer. This will be more useful. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3879) [Storage implementation] Create HDFS backing storage implementation for ATS reads
[ https://issues.apache.org/jira/browse/YARN-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-3879: - Fix Version/s: 3.1.2 3.0.3 3.2.0 > [Storage implementation] Create HDFS backing storage implementation for ATS > reads > - > > Key: YARN-3879 > URL: https://issues.apache.org/jira/browse/YARN-3879 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Tsuyoshi Ozawa >Assignee: Abhishek Modi >Priority: Major > Labels: YARN-5355 > Fix For: 3.2.0, 3.0.3, 3.1.2 > > Attachments: YARN-3879-YARN-7055.001.patch, YARN-3879.001.patch, > YARN-3879.002.patch, YARN-3879.003.patch, YARN-3879.004.patch, > YARN-3879.005.patch, YARN-3879.006.patch > > > Reader version of YARN-3841 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3879) [Storage implementation] Create HDFS backing storage implementation for ATS reads
[ https://issues.apache.org/jira/browse/YARN-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647395#comment-16647395 ] Vrushali C commented on YARN-3879: -- Patch 006 LGTM Committing shortly > [Storage implementation] Create HDFS backing storage implementation for ATS > reads > - > > Key: YARN-3879 > URL: https://issues.apache.org/jira/browse/YARN-3879 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Tsuyoshi Ozawa >Assignee: Abhishek Modi >Priority: Major > Labels: YARN-5355 > Attachments: YARN-3879-YARN-7055.001.patch, YARN-3879.001.patch, > YARN-3879.002.patch, YARN-3879.003.patch, YARN-3879.004.patch, > YARN-3879.005.patch, YARN-3879.006.patch > > > Reader version of YARN-3841 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8834) Provide Java client for fetching Yarn specific entities from TimelineReader
[
https://issues.apache.org/jira/browse/YARN-8834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647392#comment-16647392
]
Vrushali C commented on YARN-8834:
--
The patch does not apply to branch-2, looks like.
{code}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineReaderClientImpl.java:[227,13]
cannot find symbol
[ERROR] symbol: method getStatusInfo()
[ERROR] location: variable resp of type
com.sun.jersey.api.client.ClientResponse
[ERROR]
{code}
> Provide Java client for fetching Yarn specific entities from TimelineReader
> ---
>
> Key: YARN-8834
> URL: https://issues.apache.org/jira/browse/YARN-8834
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Critical
> Fix For: 3.2.0, 3.0.3, 3.1.2, 3.3.0
>
> Attachments: YARN-8834.001.patch, YARN-8834.002.patch,
> YARN-8834.003.patch, YARN-8834.004.patch, YARN-8834.005.patch,
> YARN-8834.006.patch
>
>
> While reviewing YARN-8303, we felt that it is necessary to provide
> TimelineReaderClient which wraps all the REST calls in it so that user can
> just provide application or container ids along with filters.Currently
> fetching entities from TimelineReader is only via REST call or somebody need
> to write java client get entities.
> It is good to provide TimelineReaderClient which fetch entities from
> TimelineReaderServer. This will be more useful.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
