[jira] [Created] (YARN-10299) TimeLine Service V1.5 use levelDB as backend storage will crash when data scale amount to 100GB
aimahou created YARN-10299: -- Summary: TimeLine Service V1.5 use levelDB as backend storage will crash when data scale amount to 100GB Key: YARN-10299 URL: https://issues.apache.org/jira/browse/YARN-10299 Project: Hadoop YARN Issue Type: Bug Components: timelineservice Affects Versions: 3.1.1 Reporter: aimahou h2. Issue: TimeLine Service V1.5 use levelDB as backend storage will crash when data scale amount to 100GB h2. *Specific exception:* 2020-04-24 16:06:59,914 INFO applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(691)) - No application attempt found for application_1587696012637_1143. Use a placeholder for its latest attempt id. 2020-04-24 16:06:59,914 INFO applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(691)) - No application attempt found for application_1587696012637_1143. Use a placeholder for its latest attempt id. org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: The entity for application attempt appattempt_1587696012637_1143_01 doesn't exist in the timeline store at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getApplicationAttempt(ApplicationHistoryManagerOnTimelineStore.java:183) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.generateApplicationReport(ApplicationHistoryManagerOnTimelineStore.java:677) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getApplications(ApplicationHistoryManagerOnTimelineStore.java:128) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService.getApplications(ApplicationHistoryClientService.java:195) at org.apache.hadoop.yarn.server.webapp.AppsBlock.getApplicationReport(AppsBlock.java:129) at org.apache.hadoop.yarn.server.webapp.AppsBlock.fetchData(AppsBlock.java:114) at org.apache.hadoop.yarn.server.webapp.AppsBlock.render(AppsBlock.java:137) at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69) at org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79) at org.apache.hadoop.yarn.webapp.View.render(View.java:243) at org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49) at org.apache.hadoop.yarn.webapp.hamlet2.HamletImpl$EImp._v(HamletImpl.java:117) at org.apache.hadoop.yarn.webapp.hamlet2.Hamlet$TD.__(Hamlet.java:848) at org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:71) at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82) at org.apache.hadoop.yarn.webapp.Dispatcher.render(Dispatcher.java:206) at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:165) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:304) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) at
[jira] [Created] (YARN-10298) TimeLine entity information only stored in one region when use apache HBase as backend storage
aimahou created YARN-10298:
--
Summary: TimeLine entity information only stored in one region
when use apache HBase as backend storage
Key: YARN-10298
URL: https://issues.apache.org/jira/browse/YARN-10298
Project: Hadoop YARN
Issue Type: Improvement
Components: ATSv2, timelineservice
Affects Versions: 3.1.1
Reporter: aimahou
h2. Issue
TimeLine entity information only stored in one region when use apache HBase as
backend storage
h2. Probable cause
We found in the source code that the rowKey is composed of
clusterId、userId、flowName、flowRunId and appId when hbase timeline writer stores
timeline entity info，which probably cause the rowKey is sorted by dictionary
order. Thus timeline entity may only store in one region or few adjacent
regions.
h2. Related code snippet
HBaseTimelineWriterImpl.java
public TimelineWriteResponse write(TimelineCollectorContext context,
TimelineEntities data, UserGroupInformation callerUgi)
throws IOException {
...
boolean isApplication = ApplicationEntity.isApplicationEntity(te);
byte[] rowKey;
if (isApplication) {
ApplicationRowKey applicationRowKey =
new ApplicationRowKey(clusterId, userId, flowName, flowRunId,
appId);
rowKey = applicationRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.APPLICATION_TABLE);
} else {
EntityRowKey entityRowKey =
new EntityRowKey(clusterId, userId, flowName, flowRunId, appId,
te.getType(), te.getIdPrefix(), te.getId());
rowKey = entityRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.ENTITY_TABLE);
}
if (!isApplication && SubApplicationEntity.isSubApplicationEntity(te)) {
SubApplicationRowKey subApplicationRowKey =
new SubApplicationRowKey(subApplicationUser, clusterId,
te.getType(), te.getIdPrefix(), te.getId(), userId);
rowKey = subApplicationRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.SUBAPPLICATION_TABLE);
}
...
}
h2. Suggestion
We can use the hash code of original rowKey as the rowKey to store and read
timeline entity data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10298) TimeLine entity information only stored in one region when use apache HBase as backend storage
[
https://issues.apache.org/jira/browse/YARN-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
aimahou updated YARN-10298:
---
Description:
h2. Issue
TimeLine entity information only stored in one region when use apache HBase as
backend storage
h2. Probable cause
We found in the source code that the rowKey is composed of
clusterId、userId、flowName、flowRunId and appId when hbase timeline writer stores
timeline entity info，which probably cause the rowKey is sorted by dictionary
order. Thus timeline entity may only store in one region or few adjacent
regions.
h2. Related code snippet
HBaseTimelineWriterImpl.java
{quote}public TimelineWriteResponse write(TimelineCollectorContext context,
TimelineEntities data, UserGroupInformation callerUgi)
throws IOException {
...
boolean isApplication = ApplicationEntity.isApplicationEntity(te);
byte[] rowKey;
if (isApplication)
{
ApplicationRowKey applicationRowKey = new ApplicationRowKey(clusterId, userId,
flowName, flowRunId, appId);
rowKey = applicationRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.APPLICATION_TABLE);
}
else
{
EntityRowKey entityRowKey = new EntityRowKey(clusterId, userId, flowName,
flowRunId, appId, te.getType(), te.getIdPrefix(), te.getId());
rowKey = entityRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.ENTITY_TABLE); }
if (!isApplication && SubApplicationEntity.isSubApplicationEntity(te))
{
SubApplicationRowKey subApplicationRowKey = new
SubApplicationRowKey(subApplicationUser, clusterId, te.getType(),
te.getIdPrefix(), te.getId(), userId);
rowKey = subApplicationRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.SUBAPPLICATION_TABLE);
}
...
}
{quote}
h2. Suggestion
We can use the hash code of original rowKey as the rowKey to store and read
timeline entity data.
was:
h2. Issue
TimeLine entity information only stored in one region when use apache HBase as
backend storage
h2. Probable cause
We found in the source code that the rowKey is composed of
clusterId、userId、flowName、flowRunId and appId when hbase timeline writer stores
timeline entity info，which probably cause the rowKey is sorted by dictionary
order. Thus timeline entity may only store in one region or few adjacent
regions.
h2. Related code snippet
HBaseTimelineWriterImpl.java
public TimelineWriteResponse write(TimelineCollectorContext context,
TimelineEntities data, UserGroupInformation callerUgi)
throws IOException {
...
boolean isApplication = ApplicationEntity.isApplicationEntity(te);
byte[] rowKey;
if (isApplication) {
ApplicationRowKey applicationRowKey =
new ApplicationRowKey(clusterId, userId, flowName, flowRunId,
appId);
rowKey = applicationRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.APPLICATION_TABLE);
} else {
EntityRowKey entityRowKey =
new EntityRowKey(clusterId, userId, flowName, flowRunId, appId,
te.getType(), te.getIdPrefix(), te.getId());
rowKey = entityRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.ENTITY_TABLE);
}
if (!isApplication && SubApplicationEntity.isSubApplicationEntity(te)) {
SubApplicationRowKey subApplicationRowKey =
new SubApplicationRowKey(subApplicationUser, clusterId,
te.getType(), te.getIdPrefix(), te.getId(), userId);
rowKey = subApplicationRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.SUBAPPLICATION_TABLE);
}
...
}
h2. Suggestion
We can use the hash code of original rowKey as the rowKey to store and read
timeline entity data.
> TimeLine entity information only stored in one region when use apache HBase
> as backend storage
> --
>
> Key: YARN-10298
> URL: https://issues.apache.org/jira/browse/YARN-10298
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2, timelineservice
>Affects Versions: 3.1.1
>Reporter: aimahou
>Priority: Major
>
> h2. Issue
> TimeLine entity information only stored in one region when use apache HBase
> as backend storage
> h2. Probable cause
> We found in the source code that the rowKey is composed of
> clusterId、userId、flowName、flowRunId and appId when hbase timeline writer
> stores timeline entity info，which probably cause the rowKey is sorted by
> dictionary order. Thus timeline entity may only store in one region or few
> adjacent regions.
> h2. Related code snippet
> HBaseTimelineWriterImpl.java
> {quote}public TimelineWriteResponse write(TimelineCollectorContext context,
> TimelineEntities data, UserGroupInformation callerUgi)
> throws IOException {
> ...
> boolean isApplication = ApplicationEntity.isApplicationEntity(te);
> byte[] rowKey;
> if (isApplication)
> {
> ApplicationRowKey applicationRowKey = new ApplicationRowKey(clusterId,
> userId, flowName, flowRunId, appId);
> rowKey = applicationRowKey.getRowKey();
>
[jira] [Updated] (YARN-10298) TimeLine entity information only stored in one region when use apache HBase as backend storage
[
https://issues.apache.org/jira/browse/YARN-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
aimahou updated YARN-10298:
---
Description:
h2. Issue
TimeLine entity information only stored in one region when use apache HBase as
backend storage
h2. Probable cause
We found in the source code that the rowKey is composed of
clusterId、userId、flowName、flowRunId and appId when hbase timeline writer stores
timeline entity info，which probably cause the rowKey is sorted by dictionary
order. Thus timeline entity may only store in one region or few adjacent
regions.
h2. Related code snippet
HBaseTimelineWriterImpl.java
{quote}
{code:java}
public TimelineWriteResponse write(TimelineCollectorContext context,
TimelineEntities data, UserGroupInformation callerUgi)
throws IOException {
...
boolean isApplication = ApplicationEntity.isApplicationEntity(te);
byte[] rowKey;
if (isApplication){
ApplicationRowKey applicationRowKey = new ApplicationRowKey(clusterId, userId,
flowName, flowRunId, appId); rowKey = applicationRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.APPLICATION_TABLE);
}else {
EntityRowKey entityRowKey = new EntityRowKey(clusterId, userId, flowName,
flowRunId, appId, te.getType(), te.getIdPrefix(), te.getId());
rowKey = entityRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.ENTITY_TABLE);
}
if (!isApplication && SubApplicationEntity.isSubApplicationEntity(te)) {
SubApplicationRowKey subApplicationRowKey = new
SubApplicationRowKey(subApplicationUser, clusterId, te.getType(),
te.getIdPrefix(), te.getId(), userId);
rowKey = subApplicationRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.SUBAPPLICATION_TABLE); }
...
}
{code}
{quote}
h2. Suggestion
We can use the hash code of original rowKey as the rowKey to store and read
timeline entity data.
was:
h2. Issue
TimeLine entity information only stored in one region when use apache HBase as
backend storage
h2. Probable cause
We found in the source code that the rowKey is composed of
clusterId、userId、flowName、flowRunId and appId when hbase timeline writer stores
timeline entity info，which probably cause the rowKey is sorted by dictionary
order. Thus timeline entity may only store in one region or few adjacent
regions.
h2. Related code snippet
HBaseTimelineWriterImpl.java
{quote}
{code:java}
else
public TimelineWriteResponse write(TimelineCollectorContext context,
TimelineEntities data, UserGroupInformation callerUgi)
throws IOException {
...
boolean isApplication = ApplicationEntity.isApplicationEntity(te);
byte[] rowKey;
if (isApplication){
ApplicationRowKey applicationRowKey = new ApplicationRowKey(clusterId, userId,
flowName, flowRunId, appId); rowKey = applicationRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.APPLICATION_TABLE);
}else {
EntityRowKey entityRowKey = new EntityRowKey(clusterId, userId, flowName,
flowRunId, appId, te.getType(), te.getIdPrefix(), te.getId());
rowKey = entityRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.ENTITY_TABLE);
}
if (!isApplication && SubApplicationEntity.isSubApplicationEntity(te)) {
SubApplicationRowKey subApplicationRowKey = new
SubApplicationRowKey(subApplicationUser, clusterId, te.getType(),
te.getIdPrefix(), te.getId(), userId);
rowKey = subApplicationRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.SUBAPPLICATION_TABLE); }
...
}
{code}
{quote}
h2. Suggestion
We can use the hash code of original rowKey as the rowKey to store and read
timeline entity data.
> TimeLine entity information only stored in one region when use apache HBase
> as backend storage
> --
>
> Key: YARN-10298
> URL: https://issues.apache.org/jira/browse/YARN-10298
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2, timelineservice
>Affects Versions: 3.1.1
>Reporter: aimahou
>Priority: Major
>
> h2. Issue
> TimeLine entity information only stored in one region when use apache HBase
> as backend storage
> h2. Probable cause
> We found in the source code that the rowKey is composed of
> clusterId、userId、flowName、flowRunId and appId when hbase timeline writer
> stores timeline entity info，which probably cause the rowKey is sorted by
> dictionary order. Thus timeline entity may only store in one region or few
> adjacent regions.
> h2. Related code snippet
> HBaseTimelineWriterImpl.java
> {quote}
> {code:java}
> public TimelineWriteResponse write(TimelineCollectorContext context,
> TimelineEntities data, UserGroupInformation callerUgi)
> throws IOException {
> ...
> boolean isApplication = ApplicationEntity.isApplicationEntity(te);
> byte[] rowKey;
> if (isApplication){
> ApplicationRowKey applicationRowKey = new ApplicationRowKey(clusterId,
[jira] [Updated] (YARN-10298) TimeLine entity information only stored in one region when use apache HBase as backend storage
[
https://issues.apache.org/jira/browse/YARN-10298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
aimahou updated YARN-10298:
---
Description:
h2. Issue
TimeLine entity information only stored in one region when use apache HBase as
backend storage
h2. Probable cause
We found in the source code that the rowKey is composed of
clusterId、userId、flowName、flowRunId and appId when hbase timeline writer stores
timeline entity info，which probably cause the rowKey is sorted by dictionary
order. Thus timeline entity may only store in one region or few adjacent
regions.
h2. Related code snippet
HBaseTimelineWriterImpl.java
{quote}
{code:java}
else
public TimelineWriteResponse write(TimelineCollectorContext context,
TimelineEntities data, UserGroupInformation callerUgi)
throws IOException {
...
boolean isApplication = ApplicationEntity.isApplicationEntity(te);
byte[] rowKey;
if (isApplication){
ApplicationRowKey applicationRowKey = new ApplicationRowKey(clusterId, userId,
flowName, flowRunId, appId); rowKey = applicationRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.APPLICATION_TABLE);
}else {
EntityRowKey entityRowKey = new EntityRowKey(clusterId, userId, flowName,
flowRunId, appId, te.getType(), te.getIdPrefix(), te.getId());
rowKey = entityRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.ENTITY_TABLE);
}
if (!isApplication && SubApplicationEntity.isSubApplicationEntity(te)) {
SubApplicationRowKey subApplicationRowKey = new
SubApplicationRowKey(subApplicationUser, clusterId, te.getType(),
te.getIdPrefix(), te.getId(), userId);
rowKey = subApplicationRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.SUBAPPLICATION_TABLE); }
...
}
{code}
{quote}
h2. Suggestion
We can use the hash code of original rowKey as the rowKey to store and read
timeline entity data.
was:
h2. Issue
TimeLine entity information only stored in one region when use apache HBase as
backend storage
h2. Probable cause
We found in the source code that the rowKey is composed of
clusterId、userId、flowName、flowRunId and appId when hbase timeline writer stores
timeline entity info，which probably cause the rowKey is sorted by dictionary
order. Thus timeline entity may only store in one region or few adjacent
regions.
h2. Related code snippet
HBaseTimelineWriterImpl.java
{quote}public TimelineWriteResponse write(TimelineCollectorContext context,
TimelineEntities data, UserGroupInformation callerUgi)
throws IOException {
...
boolean isApplication = ApplicationEntity.isApplicationEntity(te);
byte[] rowKey;
if (isApplication)
{
ApplicationRowKey applicationRowKey = new ApplicationRowKey(clusterId, userId,
flowName, flowRunId, appId);
rowKey = applicationRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.APPLICATION_TABLE);
}
else
{
EntityRowKey entityRowKey = new EntityRowKey(clusterId, userId, flowName,
flowRunId, appId, te.getType(), te.getIdPrefix(), te.getId());
rowKey = entityRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.ENTITY_TABLE); }
if (!isApplication && SubApplicationEntity.isSubApplicationEntity(te))
{
SubApplicationRowKey subApplicationRowKey = new
SubApplicationRowKey(subApplicationUser, clusterId, te.getType(),
te.getIdPrefix(), te.getId(), userId);
rowKey = subApplicationRowKey.getRowKey();
store(rowKey, te, flowVersion, Tables.SUBAPPLICATION_TABLE);
}
...
}
{quote}
h2. Suggestion
We can use the hash code of original rowKey as the rowKey to store and read
timeline entity data.
> TimeLine entity information only stored in one region when use apache HBase
> as backend storage
> --
>
> Key: YARN-10298
> URL: https://issues.apache.org/jira/browse/YARN-10298
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: ATSv2, timelineservice
>Affects Versions: 3.1.1
>Reporter: aimahou
>Priority: Major
>
> h2. Issue
> TimeLine entity information only stored in one region when use apache HBase
> as backend storage
> h2. Probable cause
> We found in the source code that the rowKey is composed of
> clusterId、userId、flowName、flowRunId and appId when hbase timeline writer
> stores timeline entity info，which probably cause the rowKey is sorted by
> dictionary order. Thus timeline entity may only store in one region or few
> adjacent regions.
> h2. Related code snippet
> HBaseTimelineWriterImpl.java
> {quote}
> {code:java}
> else
> public TimelineWriteResponse write(TimelineCollectorContext context,
> TimelineEntities data, UserGroupInformation callerUgi)
> throws IOException {
> ...
> boolean isApplication = ApplicationEntity.isApplicationEntity(te);
> byte[] rowKey;
> if (isApplication){
> ApplicationRowKey applicationRowKey = new ApplicationRowKey(clusterId,
> userId,
