[
https://issues.apache.org/jira/browse/ATLAS-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029258#comment-17029258
]
Zhengyi Liu commented on ATLAS-488:
-----------------------------------
Hey [~vmadugun], I am working on setting up an Atlas production env and I am
curious about the progress of this task. Seems like we didn't put extra effort
since last update and I am curious how would that block us from adopting Atlas
and eventually ingest 40 million + entities. Is there an existing benchmark
from the industry peers that prove the scalability of the existing solution? I
can see how that would give people confidence even with current approach
(single node with HA).
> Atlas service scalability
> -------------------------
>
> Key: ATLAS-488
> URL: https://issues.apache.org/jira/browse/ATLAS-488
> Project: Atlas
> Issue Type: Improvement
> Reporter: venkata madugundu
> Priority: Major
>
> Requirement -- We are looking for ways to have Atlas service running in
> multiple instances for Cloud deployment, and this requirement is kind of
> crucial to the adoption of Atlas.
> On the dev discussion forum, I had discussed earlier (Jan 25th) on the
> requirement to run multiple instances of Atlas service/runtime for horizontal
> scale. Per answers that we got from Hemanth, Atlas has a current limitation
> that only instance can be active at any given point of time for a couple of
> reasons. Few reasons cited were ...
> 1) For performance reasons, the typesystem is cached in-memory. Type lookup
> is needed for query evaluation, instance serialization/deserialization etc.,
> 2) Titan locks for Hbase transactions, causing performance degradation.
> Considering the above reasons, among them,
> #1 can be mitigated in our scenarios. The types that we create are more or
> less design-time application data-models (metamodels) and they do not change
> at runtime. So if I were to pre-register types with Atlas before my cluster
> is up, I am hoping #1 is not an issue. In the sense, how many ever Atlas
> instances come up, all would have the same state, which would mean I am
> guaranteed API correctness.
> I am not so sure how we can avoid #2. I neither understand the persistence
> mapping of metadata to the graph(Atlas) and the mapping of graph to the Hbase
> (Titan) to comment on whether #2 is really a problem for us in terms of
> lock-contention. Atlas as REST API inherently does not offer any
> "transaction" like facility to run a set of API for its end-users. Given
> that, we are more likely to implement some kind of optimistic locking
> strategy to avoid data inconsistencies.
> Are there are any other reasons, that you can think of why multiple Atlas
> runtime instances cannot be talking to a single Hbase cluster for horizontal
> scale ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)