[jira] [Updated] (ATLAS-3639) Enable to select Bigtable as a GraphStorageBackend instead of HBase
[
https://issues.apache.org/jira/browse/ATLAS-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masaru Uchida updated ATLAS-3639:
-
Summary: Enable to select Bigtable as a GraphStorageBackend instead of
HBase (was: Enable to select BigQuery as a GraphStorageBackend instead of
HBase)
> Enable to select Bigtable as a GraphStorageBackend instead of HBase
> ---
>
> Key: ATLAS-3639
> URL: https://issues.apache.org/jira/browse/ATLAS-3639
> Project: Atlas
> Issue Type: New Feature
>Reporter: Masaru Uchida
>Priority: Major
> Labels: GCP
>
> Hi, I'm developing an ETL infrastructure and researching Atlas as part of a
> metadata management infrastructure replacement.
> h2. Overview
> I want to be able to use Bigquery for graph storage backend.
> h2. Why
> * Building hbase on your own is very painful.
> * Introducing managed services lowers installation costs.
> h2. What did and did not work
> By replacing Hbase of {{HbaseStoreManager}} (atlas-janusgraph-hbase2) with
> {{BigtableConfiguration}} , operation such as startup and registration of
> Entity could be confirmed.
> {code:java}
> -this.hconf = HBaseConfiguration.create();
> +String PROJECT_ID = "my-gcp-project-id";
> +String INSTANCE_ID = "my-bigtable-instance-id";
> +String APP_PROFILE_ID = "default";
> +org.apache.hadoop.conf.Configuration conf =
> BigtableConfiguration.configure(PROJECT_ID, INSTANCE_ID);
> +Credentials credentials =
> CredentialFactory.getApplicationDefaultCredential();
> +
> +this.hconf = BigtableConfiguration.withCredentials(conf,
> credentials);
> +hconf.set(BigtableOptionsFactory.APP_PROFILE_ID_KEY, APP_PROFILE_ID);
> {code}
> https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java#L295-L303
> https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseCompat2_0.java#L48
> (this branch from {{origin/branch-2.0}} )
> However, {{Scan().SetReversed(true)}} calling in
> {{HBaseBasedAuditRepository}} fails in audit log search due to API not
> supported by Bigtable.
> bq. Reverse scans are not supported. You cannot call the method
> org.apache.hadoop.hbase.client.Scan#setReversed(boolean reversed).
> https://cloud.google.com/bigtable/docs/hbase-differences#gets_and_scans
> h2. Proposal
> * Enable to change the implementation of {{HBaseStoreManager}} if bigtable is
> specified in {{atlas.graph.storage.backend}} of atlas-application.properties.
> * It would be nice if Solr or elasticsearch could be specified in
> {{atlas.EntityAuditRepository.impl}} .
> Sorry in poor english. thank you.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3639) Enable to select Bigtable as a GraphStorageBackend instead of HBase
[
https://issues.apache.org/jira/browse/ATLAS-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masaru Uchida updated ATLAS-3639:
-
Description:
Hi, I'm developing an ETL infrastructure and researching Atlas as part of a
metadata management infrastructure replacement.
h2. Overview
I want to be able to use Bigtable for graph storage backend.
h2. Why
* Building hbase on your own is very painful.
* Introducing managed services lowers installation costs.
h2. What did and did not work
By replacing Hbase of {{HbaseStoreManager}} (atlas-janusgraph-hbase2) with
{{BigtableConfiguration}} , operation such as startup and registration of
Entity could be confirmed.
{code:java}
-this.hconf = HBaseConfiguration.create();
+String PROJECT_ID = "my-gcp-project-id";
+String INSTANCE_ID = "my-bigtable-instance-id";
+String APP_PROFILE_ID = "default";
+org.apache.hadoop.conf.Configuration conf =
BigtableConfiguration.configure(PROJECT_ID, INSTANCE_ID);
+Credentials credentials =
CredentialFactory.getApplicationDefaultCredential();
+
+this.hconf = BigtableConfiguration.withCredentials(conf, credentials);
+hconf.set(BigtableOptionsFactory.APP_PROFILE_ID_KEY, APP_PROFILE_ID);
{code}
[https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java#L295-L303]
[https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseCompat2_0.java#L48]
(this branch from {{origin/branch-2.0}} )
However, {{Scan().SetReversed(true)}} calling in {{HBaseBasedAuditRepository}}
fails in audit log search due to API not supported by Bigtable.
{quote}Reverse scans are not supported. You cannot call the method
org.apache.hadoop.hbase.client.Scan#setReversed(boolean reversed).
{quote}
[https://cloud.google.com/bigtable/docs/hbase-differences#gets_and_scans]
h2. Proposal
* Enable to change the implementation of {{HBaseStoreManager}} if bigtable is
specified in {{atlas.graph.storage.backend}} of atlas-application.properties.
* It would be nice if Solr or elasticsearch could be specified in
{{atlas.EntityAuditRepository.impl}} .
Sorry in poor english. thank you.
was:
Hi, I'm developing an ETL infrastructure and researching Atlas as part of a
metadata management infrastructure replacement.
h2. Overview
I want to be able to use Bigquery for graph storage backend.
h2. Why
* Building hbase on your own is very painful.
* Introducing managed services lowers installation costs.
h2. What did and did not work
By replacing Hbase of {{HbaseStoreManager}} (atlas-janusgraph-hbase2) with
{{BigtableConfiguration}} , operation such as startup and registration of
Entity could be confirmed.
{code:java}
-this.hconf = HBaseConfiguration.create();
+String PROJECT_ID = "my-gcp-project-id";
+String INSTANCE_ID = "my-bigtable-instance-id";
+String APP_PROFILE_ID = "default";
+org.apache.hadoop.conf.Configuration conf =
BigtableConfiguration.configure(PROJECT_ID, INSTANCE_ID);
+Credentials credentials =
CredentialFactory.getApplicationDefaultCredential();
+
+this.hconf = BigtableConfiguration.withCredentials(conf, credentials);
+hconf.set(BigtableOptionsFactory.APP_PROFILE_ID_KEY, APP_PROFILE_ID);
{code}
https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java#L295-L303
https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseCompat2_0.java#L48
(this branch from {{origin/branch-2.0}} )
However, {{Scan().SetReversed(true)}} calling in {{HBaseBasedAuditRepository}}
fails in audit log search due to API not supported by Bigtable.
bq. Reverse scans are not supported. You cannot call the method
org.apache.hadoop.hbase.client.Scan#setReversed(boolean reversed).
https://cloud.google.com/bigtable/docs/hbase-differences#gets_and_scans
h2. Proposal
* Enable to change the implementation of {{HBaseStoreManager}} if bigtable is
specified in {{atlas.graph.storage.backend}} of atlas-application.properties.
* It would be nice if Solr or elasticsearch could be specified in
{{atlas.EntityAuditRepository.impl}} .
Sorry in poor english. thank you.
> Enable to select Bigtable as a GraphStorageBackend instead of HBase
> ---
>
> Key: ATLAS-3639
> URL: https://issues.apache.org/jira/browse/ATLAS-3639
> Project: Atlas
> Issue Type: New Feature
>Reporter: Masaru Uchida
>Priority: Major
> Labels: GCP
>
> Hi,
[jira] [Updated] (ATLAS-3639) Enable to select BigQuery as a GraphStorageBackend instead of HBase
[
https://issues.apache.org/jira/browse/ATLAS-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masaru Uchida updated ATLAS-3639:
-
Issue Type: New Feature (was: Wish)
> Enable to select BigQuery as a GraphStorageBackend instead of HBase
> ---
>
> Key: ATLAS-3639
> URL: https://issues.apache.org/jira/browse/ATLAS-3639
> Project: Atlas
> Issue Type: New Feature
>Reporter: Masaru Uchida
>Priority: Major
> Labels: GCP
>
> Hi, I'm developing an ETL infrastructure and researching Atlas as part of a
> metadata management infrastructure replacement.
> h2. Overview
> I want to be able to use Bigquery for graph storage backend.
> h2. Why
> * Building hbase on your own is very painful.
> * Introducing managed services lowers installation costs.
> h2. What did and did not work
> By replacing Hbase of {{HbaseStoreManager}} (atlas-janusgraph-hbase2) with
> {{BigtableConfiguration}} , operation such as startup and registration of
> Entity could be confirmed.
> {code:java}
> -this.hconf = HBaseConfiguration.create();
> +String PROJECT_ID = "my-gcp-project-id";
> +String INSTANCE_ID = "my-bigtable-instance-id";
> +String APP_PROFILE_ID = "default";
> +org.apache.hadoop.conf.Configuration conf =
> BigtableConfiguration.configure(PROJECT_ID, INSTANCE_ID);
> +Credentials credentials =
> CredentialFactory.getApplicationDefaultCredential();
> +
> +this.hconf = BigtableConfiguration.withCredentials(conf,
> credentials);
> +hconf.set(BigtableOptionsFactory.APP_PROFILE_ID_KEY, APP_PROFILE_ID);
> {code}
> https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java#L295-L303
> https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseCompat2_0.java#L48
> (this branch from {{origin/branch-2.0}} )
> However, {{Scan().SetReversed(true)}} calling in
> {{HBaseBasedAuditRepository}} fails in audit log search due to API not
> supported by Bigtable.
> bq. Reverse scans are not supported. You cannot call the method
> org.apache.hadoop.hbase.client.Scan#setReversed(boolean reversed).
> https://cloud.google.com/bigtable/docs/hbase-differences#gets_and_scans
> h2. Proposal
> * Enable to change the implementation of {{HBaseStoreManager}} if bigtable is
> specified in {{atlas.graph.storage.backend}} of atlas-application.properties.
> * It would be nice if Solr or elasticsearch could be specified in
> {{atlas.EntityAuditRepository.impl}} .
> Sorry in poor english. thank you.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ATLAS-3639) Enable to select BigQuery as a GraphStorageBackend instead of HBase
Masaru Uchida created ATLAS-3639:
Summary: Enable to select BigQuery as a GraphStorageBackend
instead of HBase
Key: ATLAS-3639
URL: https://issues.apache.org/jira/browse/ATLAS-3639
Project: Atlas
Issue Type: Wish
Reporter: Masaru Uchida
Hi, I'm developing an ETL infrastructure and researching Atlas as part of a
metadata management infrastructure replacement.
h2. Overview
I want to be able to use Bigquery for graph storage backend.
h2. Why
* Building hbase on your own is very painful.
* Introducing managed services lowers installation costs.
h2. What did and did not work
By replacing Hbase of {{HbaseStoreManager}} (atlas-janusgraph-hbase2) with
{{BigtableConfiguration}} , operation such as startup and registration of
Entity could be confirmed.
{code:java}
-this.hconf = HBaseConfiguration.create();
+String PROJECT_ID = "my-gcp-project-id";
+String INSTANCE_ID = "my-bigtable-instance-id";
+String APP_PROFILE_ID = "default";
+org.apache.hadoop.conf.Configuration conf =
BigtableConfiguration.configure(PROJECT_ID, INSTANCE_ID);
+Credentials credentials =
CredentialFactory.getApplicationDefaultCredential();
+
+this.hconf = BigtableConfiguration.withCredentials(conf, credentials);
+hconf.set(BigtableOptionsFactory.APP_PROFILE_ID_KEY, APP_PROFILE_ID);
{code}
https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java#L295-L303
https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseCompat2_0.java#L48
(this branch from {{origin/branch-2.0}} )
However, {{Scan().SetReversed(true)}} calling in {{HBaseBasedAuditRepository}}
fails in audit log search due to API not supported by Bigtable.
bq. Reverse scans are not supported. You cannot call the method
org.apache.hadoop.hbase.client.Scan#setReversed(boolean reversed).
https://cloud.google.com/bigtable/docs/hbase-differences#gets_and_scans
h2. Proposal
* Enable to change the implementation of {{HBaseStoreManager}} if bigtable is
specified in {{atlas.graph.storage.backend}} of atlas-application.properties.
* It would be nice if Solr or elasticsearch could be specified in
{{atlas.EntityAuditRepository.impl}} .
Sorry in poor english. thank you.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
