[jira] [Updated] (YARN-9374) HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage down
[ https://issues.apache.org/jira/browse/YARN-9374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9374: Affects Version/s: 3.2.0 > HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage > down > > > Key: YARN-9374 > URL: https://issues.apache.org/jira/browse/YARN-9374 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > > HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage > is down. Currently we check if hbase storage is down in TimelineReader before > reading entities and fail immediately in YARN-8302. Similar fix is needed for > write. Async is handled in YARN-9335. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9374) HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage down
[ https://issues.apache.org/jira/browse/YARN-9374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9374: Component/s: ATSv2 > HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage > down > > > Key: YARN-9374 > URL: https://issues.apache.org/jira/browse/YARN-9374 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > > HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage > is down. Currently we check if hbase storage is down in TimelineReader before > reading entities and fail immediately in YARN-8302. Similar fix is needed for > write. Async is handled in YARN-9335. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9374) HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage down
[ https://issues.apache.org/jira/browse/YARN-9374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9374: Issue Type: Sub-task (was: Bug) Parent: YARN-7055 > HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage > down > > > Key: YARN-9374 > URL: https://issues.apache.org/jira/browse/YARN-9374 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > > HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage > is down. Currently we check if hbase storage is down in TimelineReader before > reading entities and fail immediately in YARN-8302. Similar fix is needed for > write. Async is handled in YARN-9335. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9374) HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage down
Prabhu Joseph created YARN-9374: --- Summary: HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage down Key: YARN-9374 URL: https://issues.apache.org/jira/browse/YARN-9374 Project: Hadoop YARN Issue Type: Bug Reporter: Prabhu Joseph Assignee: Prabhu Joseph HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage is down. Currently we check if hbase storage is down in TimelineReader before reading entities and fail immediately in YARN-8302. Similar fix is needed for write. Async is handled in YARN-9335. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9335) [atsv2] Restrict the number of elements held in NM timeline collector when backend is unreachable for async calls
[ https://issues.apache.org/jira/browse/YARN-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-9335: Summary: [atsv2] Restrict the number of elements held in NM timeline collector when backend is unreachable for async calls (was: [atsv2] Restrict the number of elements held in NM timeline collector when backend is unreachable for asycn calls) > [atsv2] Restrict the number of elements held in NM timeline collector when > backend is unreachable for async calls > - > > Key: YARN-9335 > URL: https://issues.apache.org/jira/browse/YARN-9335 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vrushali C >Assignee: Abhishek Modi >Priority: Major > > For ATSv2 , if the backend is unreachable, the number/size of data held in > timeline collector's memory increases significantly. This is not good for the > NM memory. > Filing jira to set a limit on how many/much should be retained by the > timeline collector in memory in case the backend is not reachable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9335) [atsv2] Restrict the number of elements held in NM timeline collector when backend is unreachable for asycn calls
[ https://issues.apache.org/jira/browse/YARN-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788667#comment-16788667 ] Abhishek Modi commented on YARN-9335: - Sure [~Prabhu Joseph]. You can take over sync writes part. I will attach a patch for async one. Thanks. > [atsv2] Restrict the number of elements held in NM timeline collector when > backend is unreachable for asycn calls > - > > Key: YARN-9335 > URL: https://issues.apache.org/jira/browse/YARN-9335 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vrushali C >Assignee: Abhishek Modi >Priority: Major > > For ATSv2 , if the backend is unreachable, the number/size of data held in > timeline collector's memory increases significantly. This is not good for the > NM memory. > Filing jira to set a limit on how many/much should be retained by the > timeline collector in memory in case the backend is not reachable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9335) [atsv2] Restrict the number of elements held in NM timeline collector when backend is unreachable for asycn calls
[ https://issues.apache.org/jira/browse/YARN-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788662#comment-16788662 ] Prabhu Joseph commented on YARN-9335: - [~abmodi] If you are fine, will work on sync writes part with ensuring storage is not down. > [atsv2] Restrict the number of elements held in NM timeline collector when > backend is unreachable for asycn calls > - > > Key: YARN-9335 > URL: https://issues.apache.org/jira/browse/YARN-9335 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vrushali C >Assignee: Abhishek Modi >Priority: Major > > For ATSv2 , if the backend is unreachable, the number/size of data held in > timeline collector's memory increases significantly. This is not good for the > NM memory. > Filing jira to set a limit on how many/much should be retained by the > timeline collector in memory in case the backend is not reachable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9303) Username splits won't help timelineservice.app_flow table
[
https://issues.apache.org/jira/browse/YARN-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788657#comment-16788657
]
Prabhu Joseph commented on YARN-9303:
-
1. Removed the {{userName}} Pre-Splits from {{timelineservice.app_flow}} hbase
table and left with Auto-Splits
*Reason:*
The current rowkey starts with inverted timestamp (eg: , 9998, 9997)
from {{application_id}} for which the pre-splits cannot help. A hash value
before the rowkey can prevent hotspotting but which will require a complex
logic and a sort during fetch to display the apps ordered. This table won't
have much load as we insert a row for every app submission, so HotSpotting
won't be much a problem.
Auto Splitting should be good. {{flowRun}} and {{flowActivity}} also uses the
Auto Splits.
2. Removed {{KeyPrefixRegionSplitPolicy}} as the table does not have any groups
based on prefix. The default {{IncreasingToUpperBoundRegionSplitPolicy}} will
work fine.
Have reviewed other 6 tables which are looking fine except one problem when
{{cluster_id}} for {{application, domain}} table does not start with lower-case
letter as the pre-splits logic is based on lower-case. Allowing user to
configure the pre-splits based on their {{cluster_id}} and {{userName}} will
fix this issue. Reported YARN-9373 for the same.
[~rohithsharma], [~vrushalic] Can you review this Jira when you get time.
> Username splits won't help timelineservice.app_flow table
> -
>
> Key: YARN-9303
> URL: https://issues.apache.org/jira/browse/YARN-9303
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.2
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: Only_Last_Region_Used.png, YARN-9303-001.patch
>
>
> timelineservice.app_flow hbase table uses pre split logic based on username
> whereas the rowkeys starts with inverted timestamp (Long.MAX_VALUE - ts). All
> data will go to the last region and remaining regions will never be inserted.
> Need to choose right split or use auto-split.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9303) Username splits won't help timelineservice.app_flow table
[
https://issues.apache.org/jira/browse/YARN-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788646#comment-16788646
]
Hadoop QA commented on YARN-9303:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} YARN-9303 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9303 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23669/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Username splits won't help timelineservice.app_flow table
> -
>
> Key: YARN-9303
> URL: https://issues.apache.org/jira/browse/YARN-9303
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.2
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: Only_Last_Region_Used.png, YARN-9303-001.patch
>
>
> timelineservice.app_flow hbase table uses pre split logic based on username
> whereas the rowkeys starts with inverted timestamp (Long.MAX_VALUE - ts). All
> data will go to the last region and remaining regions will never be inserted.
> Need to choose right split or use auto-split.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9303) Username splits won't help timelineservice.app_flow table
[ https://issues.apache.org/jira/browse/YARN-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9303: Attachment: Only_Last_Region_Used.png > Username splits won't help timelineservice.app_flow table > - > > Key: YARN-9303 > URL: https://issues.apache.org/jira/browse/YARN-9303 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: Only_Last_Region_Used.png, YARN-9303-001.patch > > > timelineservice.app_flow hbase table uses pre split logic based on username > whereas the rowkeys starts with inverted timestamp (Long.MAX_VALUE - ts). All > data will go to the last region and remaining regions will never be inserted. > Need to choose right split or use auto-split. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9303) Username splits won't help timelineservice.app_flow table
[ https://issues.apache.org/jira/browse/YARN-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9303: Attachment: YARN-9303-001.patch > Username splits won't help timelineservice.app_flow table > - > > Key: YARN-9303 > URL: https://issues.apache.org/jira/browse/YARN-9303 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9303-001.patch > > > timelineservice.app_flow hbase table uses pre split logic based on username > whereas the rowkeys starts with inverted timestamp (Long.MAX_VALUE - ts). All > data will go to the last region and remaining regions will never be inserted. > Need to choose right split or use auto-split. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9303) Username splits won't help timelineservice.app_flow table
[ https://issues.apache.org/jira/browse/YARN-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9303: Description: timelineservice.app_flow hbase table uses pre split logic based on username whereas the rowkeys starts with inverted timestamp (Long.MAX_VALUE - ts). All data will go to the last region and remaining regions will never be inserted. Need to choose right split or use auto-split. was: application, domain -> uses clusterId splits entity , subapplication-> uses username splits flowactivity, flowrun -> no splits (starts with clusterId) app_flow -> make no splits > Username splits won't help timelineservice.app_flow table > - > > Key: YARN-9303 > URL: https://issues.apache.org/jira/browse/YARN-9303 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > > timelineservice.app_flow hbase table uses pre split logic based on username > whereas the rowkeys starts with inverted timestamp (Long.MAX_VALUE - ts). All > data will go to the last region and remaining regions will never be inserted. > Need to choose right split or use auto-split. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9303) Username splits won't help timelineservice.app_flow table
[ https://issues.apache.org/jira/browse/YARN-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9303: Description: application, domain -> uses clusterId splits entity , subapplication-> uses username splits flowactivity, flowrun -> no splits (starts with clusterId) app_flow -> make no splits was: Currently Timeline Service Storage tables uses splits based on username. ApplicationTable rowkey starts with ClusterId, AppToFlowTable rowkey starts with AppId won;t work well with username splits. The Hbase write will hotspot to one RegionServer. Need to optimize the pre split keys for each tables based on the RowKey. http://archive.cloudera.com/cdh5/cdh/5/hbase-0.98.6-cdh5.3.8/book/rowkey.design.html > Username splits won't help timelineservice.app_flow table > - > > Key: YARN-9303 > URL: https://issues.apache.org/jira/browse/YARN-9303 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > > application, domain -> uses clusterId splits > entity , subapplication-> uses username splits > flowactivity, flowrun -> no splits (starts with clusterId) > app_flow -> make no splits -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9303) Username splits won't help timelineservice.app_flow table
[ https://issues.apache.org/jira/browse/YARN-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9303: Summary: Username splits won't help timelineservice.app_flow table (was: Optimizing Pre Split keys for Hbase Timeline Service Storage tables) > Username splits won't help timelineservice.app_flow table > - > > Key: YARN-9303 > URL: https://issues.apache.org/jira/browse/YARN-9303 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.2 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > > Currently Timeline Service Storage tables uses splits based on username. > ApplicationTable rowkey starts with ClusterId, AppToFlowTable rowkey starts > with AppId won;t work well with username splits. The Hbase write will hotspot > to one RegionServer. Need to optimize the pre split keys for each tables > based on the RowKey. > http://archive.cloudera.com/cdh5/cdh/5/hbase-0.98.6-cdh5.3.8/book/rowkey.design.html -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9373) HBaseTimelineSchemaCreator has to allow user to configure pre-splits
Prabhu Joseph created YARN-9373: --- Summary: HBaseTimelineSchemaCreator has to allow user to configure pre-splits Key: YARN-9373 URL: https://issues.apache.org/jira/browse/YARN-9373 Project: Hadoop YARN Issue Type: Bug Components: ATSv2 Affects Versions: 3.2.0 Reporter: Prabhu Joseph Assignee: Prabhu Joseph Most of the TimelineService HBase tables is set with username splits which is based on lowercase alphabet (a,ad,an,b,ca). This won't help if the rowkey starts with either number or uppercase alphabet. We need to allow user to configure based upon their data. For example, say a user has configured the yarn.resourcemanager.cluster-id to be ATS or 123, then the splits can be configured as A,B,C,,, or 100,200,300,,, -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
