[
https://issues.apache.org/jira/browse/ATLAS-2270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269246#comment-16269246
]
Pierre Padovani edited comment on ATLAS-2270 at 11/28/17 6:48 PM:
------------------------------------------------------------------
Hi [~grahamwallis]
IMHO - In terms of the Janus-ES topology support:
* Fix the janus pom.xml and include the es rest client dependency and ship that
with the atlas distribution. Otherwise implementors will have to copy that jar
into the lib of Atlas before starting.
* Require anyone wanting to use a janus-es topology to stand up their own es
cluster and supply the hostnames in the atlas properties. This isn't all that
hard: download ES then start it (for a single node config).
* Include a docker file (I'll gladly contribute mine as an alternative to the
current one.) that provides this topology for development/example purposes. (We
exclusively use this docker for dev/debug at this point, it is just a single
Atlas node.)
We are in the middle of our development cycle in terms of metadata support, and
our infrastructure lacks any Hadoop support, and will likely not contain it in
the future. As we evaluated possible metadata products, Atlas stood out,
however using Titan was not something we wanted. Seeing the move to Janus was
huge for us, we intend to deploy Atlas Janus 0.2.0 on Cassandra + ES.
*EDIT* To be clear I implemented all of the above changes, and have built and
deployed an Atlas docker container running Janus 0.2.0 using Cassandra and
Elasticsearch. As far as we can tell it works very well.
Thanks!
Pierre
was (Author: ppadovani):
Hi [~grahamwallis]
IMHO - In terms of the Janus-ES topology support:
* Fix the janus pom.xml and include the es rest client dependency and ship that
with the atlas distribution. Otherwise implementors will have to copy that jar
into the lib of Atlas before starting.
* Require anyone wanting to use a janus-es topology to stand up their own es
cluster and supply the hostnames in the atlas properties. This isn't all that
hard: download ES then start it (for a single node config).
* Include a docker file (I'll gladly contribute mine as an alternative to the
current one.) that provides this topology for development/example purposes. (We
exclusively use this docker for dev/debug at this point, it is just a single
Atlas node.)
We are in the middle of our development cycle in terms of metadata support, and
our infrastructure lacks any Hadoop support, and will likely not contain it in
the future. As we evaluated possible metadata products, Atlas stood out,
however using Titan was not something we wanted. Seeing the move to Janus was
huge for us, we intend to deploy Atlas Janus 0.2.0 on Cassandra + ES.
Thanks!
Pierre
> Supported combinations of persistent store and index backend
> ------------------------------------------------------------
>
> Key: ATLAS-2270
> URL: https://issues.apache.org/jira/browse/ATLAS-2270
> Project: Atlas
> Issue Type: Bug
> Reporter: Graham Wallis
>
> We need to discuss and decide which combinations of persistent store and
> indexing backend Atlas 1.0.0 (master) should support. This includes
> building/running Atlas as a standalone package and running UTs/ITs as part of
> the Atlas build.
> This JIRA focusses on titan0 and janusgraph 0.2.0, as they are the graph
> databases that will be supported in master/1.0.0. This JIRA deliberately
> ignores titan1 and janusgraph 0.1.1 as the former should be
> deprecated/removed and the other is a transient state as we get to janusgraph
> 0.2.0.
> With titan0 as the graph provider, Atlas has supported the following
> combinations of persistent store and indexer. It is suggested that this set
> is kept unchanged:
> {{
> titan0 solr es
> ------------------------------------
> berkeley 0 1
> hbase 1 0
> cassandra 0 0
> }}
> With janusgraph (0.2.0) as the graph provider, Atlas *could* support
> additional combinations. Cassandra is included in this discussion pending
> response to ATLAS-2259.
> {{
> janus 0.2.0 solr es
> ------------------------------------
> berkeley ? 1
> hbase 1 ?
> cassandra ? ?
> }}
> It is suggested that the combinations marked with '1' should be continued and
> the remaining 4 combinations, marked with '?', should be considered. There
> seems to be evidence of people using all 4 of these combinations, although
> not necessarily with Atlas.
> Depending on the decision made above, we need to ensure that it is possible
> to build Atlas as a standalone package with any of the combinations - i.e.
> that they are mutually exclusive and do not interfere with one another. They
> currently interfere which makes it impossible to build Atlas with
> -Pdist,berkeley-elasticsearch because the 'dist' profile will exclude jars
> that are needed by the berkeley-elasticsearch profile - which leads to class
> not found exceptions when the Atlas server is started. The solution to this
> could be very simple, or slightly more sophisticated, depending on how many
> of the combinations we choose to support.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
