[
https://issues.apache.org/jira/browse/ATLAS-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masaru Uchida updated ATLAS-3639:
---------------------------------
Description:
Hi, I'm developing an ETL infrastructure and researching Atlas as part of a
metadata management infrastructure replacement.
h2. Overview
I want to be able to use Bigtable for graph storage backend.
h2. Why
* Building hbase on your own is very painful.
* Introducing managed services lowers installation costs.
h2. What did and did not work
By replacing Hbase of {{HbaseStoreManager}} (atlas-janusgraph-hbase2) with
{{BigtableConfiguration}} , operation such as startup and registration of
Entity could be confirmed.
{code:java}
- this.hconf = HBaseConfiguration.create();
+ String PROJECT_ID = "my-gcp-project-id";
+ String INSTANCE_ID = "my-bigtable-instance-id";
+ String APP_PROFILE_ID = "default";
+ org.apache.hadoop.conf.Configuration conf =
BigtableConfiguration.configure(PROJECT_ID, INSTANCE_ID);
+ Credentials credentials =
CredentialFactory.getApplicationDefaultCredential();
+
+ this.hconf = BigtableConfiguration.withCredentials(conf, credentials);
+ hconf.set(BigtableOptionsFactory.APP_PROFILE_ID_KEY, APP_PROFILE_ID);
{code}
[https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java#L295-L303]
[https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseCompat2_0.java#L48]
(this branch from {{origin/branch-2.0}} )
However, {{Scan().SetReversed(true)}} calling in {{HBaseBasedAuditRepository}}
fails in audit log search due to API not supported by Bigtable.
{quote}Reverse scans are not supported. You cannot call the method
org.apache.hadoop.hbase.client.Scan#setReversed(boolean reversed).
{quote}
[https://cloud.google.com/bigtable/docs/hbase-differences#gets_and_scans]
h2. Proposal
* Enable to change the implementation of {{HBaseStoreManager}} if bigtable is
specified in {{atlas.graph.storage.backend}} of atlas-application.properties.
* It would be nice if Solr or elasticsearch could be specified in
{{atlas.EntityAuditRepository.impl}} .
Sorry in poor english. thank you.
was:
Hi, I'm developing an ETL infrastructure and researching Atlas as part of a
metadata management infrastructure replacement.
h2. Overview
I want to be able to use Bigquery for graph storage backend.
h2. Why
* Building hbase on your own is very painful.
* Introducing managed services lowers installation costs.
h2. What did and did not work
By replacing Hbase of {{HbaseStoreManager}} (atlas-janusgraph-hbase2) with
{{BigtableConfiguration}} , operation such as startup and registration of
Entity could be confirmed.
{code:java}
- this.hconf = HBaseConfiguration.create();
+ String PROJECT_ID = "my-gcp-project-id";
+ String INSTANCE_ID = "my-bigtable-instance-id";
+ String APP_PROFILE_ID = "default";
+ org.apache.hadoop.conf.Configuration conf =
BigtableConfiguration.configure(PROJECT_ID, INSTANCE_ID);
+ Credentials credentials =
CredentialFactory.getApplicationDefaultCredential();
+
+ this.hconf = BigtableConfiguration.withCredentials(conf, credentials);
+ hconf.set(BigtableOptionsFactory.APP_PROFILE_ID_KEY, APP_PROFILE_ID);
{code}
https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java#L295-L303
https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseCompat2_0.java#L48
(this branch from {{origin/branch-2.0}} )
However, {{Scan().SetReversed(true)}} calling in {{HBaseBasedAuditRepository}}
fails in audit log search due to API not supported by Bigtable.
bq. Reverse scans are not supported. You cannot call the method
org.apache.hadoop.hbase.client.Scan#setReversed(boolean reversed).
https://cloud.google.com/bigtable/docs/hbase-differences#gets_and_scans
h2. Proposal
* Enable to change the implementation of {{HBaseStoreManager}} if bigtable is
specified in {{atlas.graph.storage.backend}} of atlas-application.properties.
* It would be nice if Solr or elasticsearch could be specified in
{{atlas.EntityAuditRepository.impl}} .
Sorry in poor english. thank you.
> Enable to select Bigtable as a GraphStorageBackend instead of HBase
> -------------------------------------------------------------------
>
> Key: ATLAS-3639
> URL: https://issues.apache.org/jira/browse/ATLAS-3639
> Project: Atlas
> Issue Type: New Feature
> Reporter: Masaru Uchida
> Priority: Major
> Labels: GCP
>
> Hi, I'm developing an ETL infrastructure and researching Atlas as part of a
> metadata management infrastructure replacement.
> h2. Overview
> I want to be able to use Bigtable for graph storage backend.
> h2. Why
> * Building hbase on your own is very painful.
> * Introducing managed services lowers installation costs.
> h2. What did and did not work
> By replacing Hbase of {{HbaseStoreManager}} (atlas-janusgraph-hbase2) with
> {{BigtableConfiguration}} , operation such as startup and registration of
> Entity could be confirmed.
> {code:java}
> - this.hconf = HBaseConfiguration.create();
> + String PROJECT_ID = "my-gcp-project-id";
> + String INSTANCE_ID = "my-bigtable-instance-id";
> + String APP_PROFILE_ID = "default";
> + org.apache.hadoop.conf.Configuration conf =
> BigtableConfiguration.configure(PROJECT_ID, INSTANCE_ID);
> + Credentials credentials =
> CredentialFactory.getApplicationDefaultCredential();
> +
> + this.hconf = BigtableConfiguration.withCredentials(conf,
> credentials);
> + hconf.set(BigtableOptionsFactory.APP_PROFILE_ID_KEY, APP_PROFILE_ID);
> {code}
> [https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java#L295-L303]
> [https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseCompat2_0.java#L48]
> (this branch from {{origin/branch-2.0}} )
> However, {{Scan().SetReversed(true)}} calling in
> {{HBaseBasedAuditRepository}} fails in audit log search due to API not
> supported by Bigtable.
> {quote}Reverse scans are not supported. You cannot call the method
> org.apache.hadoop.hbase.client.Scan#setReversed(boolean reversed).
> {quote}
> [https://cloud.google.com/bigtable/docs/hbase-differences#gets_and_scans]
> h2. Proposal
> * Enable to change the implementation of {{HBaseStoreManager}} if bigtable
> is specified in {{atlas.graph.storage.backend}} of
> atlas-application.properties.
> * It would be nice if Solr or elasticsearch could be specified in
> {{atlas.EntityAuditRepository.impl}} .
> Sorry in poor english. thank you.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
