[
https://issues.apache.org/jira/browse/YARN-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-3411:
-----------------------------
Attachment: YARN-3411.poc.6.txt
Uploading a patch with the review suggestions. No major changes, some coding
updates.
Also, rebased to pull in latest commits (Phoenix related changes). But now I am
having some trouble getting the timelineservice module to build since it
includes both the phoenix and hbase dependencies in the pom. I get "Some
Enforcer rules have failed" errors. I am working on resolving those. Any more
eyes on this build error would help!
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-enforcer-plugin:1.3.1:enforce (depcheck) on
project hadoop-yarn-server-timelineservice: Some Enforcer rules have failed.
Look above for specific messages explaining why the rule failed. -> [Help 1]
[ERROR]
{code}
{code}
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @
hadoop-yarn-server-timelineservice ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @
hadoop-yarn-server-timelineservice ---
[WARNING]
Dependency convergence error for org.apache.hbase:hbase-common:1.0.1 paths to
dependency are:
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-client:1.0.1
+-org.apache.hbase:hbase-common:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-common:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-common:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-common:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-common:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-prefix-tree:0.98.9-hadoop2
+-org.apache.hbase:hbase-common:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-prefix-tree:0.98.9-hadoop2
+-org.apache.hbase:hbase-common:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-common:0.98.9-hadoop2
[WARNING]
Dependency convergence error for org.apache.hbase:hbase-protocol:1.0.1 paths to
dependency are:
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-client:1.0.1
+-org.apache.hbase:hbase-protocol:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-protocol:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-protocol:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-protocol:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-protocol:0.98.9-hadoop2
[WARNING]
Dependency convergence error for org.apache.hbase:hbase-hadoop-compat:1.0.1
paths to dependency are:
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-hadoop-compat:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-hadoop-compat:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-hadoop2-compat:1.0.1
+-org.apache.hbase:hbase-hadoop-compat:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-prefix-tree:0.98.9-hadoop2
+-org.apache.hbase:hbase-hadoop-compat:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-hadoop-compat:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-hadoop-compat:0.98.9-hadoop2
[WARNING]
Dependency convergence error for org.apache.hbase:hbase-client:1.0.1 paths to
dependency are:
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-client:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-client:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-client:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-client:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-client:0.98.9-hadoop2
[WARNING]
Dependency convergence error for org.apache.hbase:hbase-hadoop2-compat:1.0.1
paths to dependency are:
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-hadoop2-compat:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-hadoop2-compat:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-prefix-tree:0.98.9-hadoop2
+-org.apache.hbase:hbase-hadoop2-compat:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-hadoop2-compat:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-hadoop2-compat:0.98.9-hadoop2
[WARNING]
Dependency convergence error for org.apache.commons:commons-math:2.2 paths to
dependency are:
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-hadoop-compat:1.0.1
+-org.apache.commons:commons-math:2.2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-hadoop2-compat:1.0.1
+-org.apache.commons:commons-math:2.2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.commons:commons-math:2.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.commons:commons-math:2.1
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence
failed with message:
Failed while enforcing releasability the error(s) are [
Dependency convergence error for org.apache.hbase:hbase-common:1.0.1 paths to
dependency are:
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-client:1.0.1
+-org.apache.hbase:hbase-common:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-common:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-common:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-common:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-common:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-prefix-tree:0.98.9-hadoop2
+-org.apache.hbase:hbase-common:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-prefix-tree:0.98.9-hadoop2
+-org.apache.hbase:hbase-common:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-common:0.98.9-hadoop2
,
Dependency convergence error for org.apache.hbase:hbase-protocol:1.0.1 paths to
dependency are:
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-client:1.0.1
+-org.apache.hbase:hbase-protocol:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-protocol:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-protocol:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-protocol:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-protocol:0.98.9-hadoop2
,
Dependency convergence error for org.apache.hbase:hbase-hadoop-compat:1.0.1
paths to dependency are:
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-hadoop-compat:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-hadoop-compat:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-hadoop2-compat:1.0.1
+-org.apache.hbase:hbase-hadoop-compat:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-prefix-tree:0.98.9-hadoop2
+-org.apache.hbase:hbase-hadoop-compat:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-hadoop-compat:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-hadoop-compat:0.98.9-hadoop2
,
Dependency convergence error for org.apache.hbase:hbase-client:1.0.1 paths to
dependency are:
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-client:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-client:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-client:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-client:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-client:0.98.9-hadoop2
,
Dependency convergence error for org.apache.hbase:hbase-hadoop2-compat:1.0.1
paths to dependency are:
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-hadoop2-compat:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-hadoop2-compat:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-prefix-tree:0.98.9-hadoop2
+-org.apache.hbase:hbase-hadoop2-compat:1.0.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-hadoop2-compat:0.98.9-hadoop2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.hbase:hbase-hadoop2-compat:0.98.9-hadoop2
,
Dependency convergence error for org.apache.commons:commons-math:2.2 paths to
dependency are:
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-hadoop-compat:1.0.1
+-org.apache.commons:commons-math:2.2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.hbase:hbase-testing-util:1.0.1
+-org.apache.hbase:hbase-hadoop2-compat:1.0.1
+-org.apache.commons:commons-math:2.2
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.commons:commons-math:2.1
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT
+-org.apache.phoenix:phoenix-core:4.3.0
+-org.apache.hbase:hbase-server:1.0.1
+-org.apache.commons:commons-math:2.1
]
[INFO]
{code}
> [Storage implementation] explore the native HBase write schema for storage
> --------------------------------------------------------------------------
>
> Key: YARN-3411
> URL: https://issues.apache.org/jira/browse/YARN-3411
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Sangjin Lee
> Assignee: Vrushali C
> Priority: Critical
> Attachments: ATSv2BackendHBaseSchemaproposal.pdf,
> YARN-3411.poc.2.txt, YARN-3411.poc.3.txt, YARN-3411.poc.4.txt,
> YARN-3411.poc.5.txt, YARN-3411.poc.6.txt, YARN-3411.poc.txt
>
>
> There is work that's in progress to implement the storage based on a Phoenix
> schema (YARN-3134).
> In parallel, we would like to explore an implementation based on a native
> HBase schema for the write path. Such a schema does not exclude using
> Phoenix, especially for reads and offline queries.
> Once we have basic implementations of both options, we could evaluate them in
> terms of performance, scalability, usability, etc. and make a call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
