This is an automatically generated e-mail. To reply, visit:

(Updated April 13, 2018, 2:55 a.m.)

Review request for atlas, David Radley, Madhan Neethiraj, and Sarath 


This update is a rebase against master. It also resolves an inconsistent 
startup of elasticsearch.

This partially resolves some of the issues found in this review, but I believe 
further discussion is needed to resolve things correctly.


  - Berkley db is excluded from the distribution, so there is this copy hack 
that must happen. This has always been the case and falls outside the scope of 
this patch
  - The way I configure the Constants is not ideal. I am in need of guidance on 
where/how to implement this fix as my understanding of the internals of Atlas 
is limited.
  - The connection to elasticsearch is now done by REST and not by native 
client. The jars do not contain any shaded 3rd party jars, and are needed for 
the REST client. If we build/distribute a full distribution that isn't based on 
a particular profile, these jars will be required if someone wants to use 
Elasticsearch. They should be harmless due to the lack of shaded 3rd party jars.
  - I need to add some documentation around using this profile, but wish to 
hold off until we have agreement on what and how to deal with the Elasticsearch 

To Run this profile:
mvn clean install -Pdist,berkeley-elasticsearch -DskipTests
cd distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT
./bin/atlas_stop.py  <-- Atlas will NOT start correctly becuase the berkley db 
jar is missing, starting and stopping will create the lib dir where the berkley 
db jar is needed.

Bugs: ATLAS-2478

Repository: atlas


This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates 

Included with this patch is an update to the berkley-elasticsearch profile to 
automatically download and include elasticsearch as a side application much 
like solr is. Updates to the start/stop/conf scripts are included as well.

NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to 
/atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There 
are six constants that are incorrectly named with a '.' (dot). This is not 
supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the 
field names can be collectively thought of as an object. In the case of the 
fields defined in the Constants.java file, 'type' is defined as a string field, 
and 'type.name' is also defined as a string field. Elasticsearch sees this as 
an error, since it cannot convert type to an object. The fix included simply 
changes the field names from using a '.' (dot) to an '_' (underscore). This 
should NOT affect compatibility with hbase/solr for new installs. For existing 
installations, a reindex will be required as the field names will have changed.

**Query**: There is a way we can simplify integration/unit tests for the 
in-memory graph store by using a maven plugin that will download and run an 
elasticsearch node. This is nothing more than a maven change, and change to the 
atlas-application.properties to switch to elasticsearch from solr. I did not 
implement this, but am curious if this change would be desired. If so, this can 
be done with a separate ticket.

Diffs (updated)

  common/src/main/java/org/apache/atlas/repository/Constants.java 01e49157d 
  distro/pom.xml 1f4c6d557 
  distro/src/bin/atlas_config.py 9062da649 
  distro/src/bin/atlas_start.py 61d69eb21 
  distro/src/bin/atlas_stop.py 94c3d6d46 
  distro/src/conf/atlas-env.sh 9213f488b 
  distro/src/main/assemblies/standalone-package.xml dc2a66ba9 
  docs/src/site/twiki/Configuration.twiki 63c3fce96 
  docs/src/site/twiki/HighAvailability.twiki 4270d0974 
  docs/src/site/twiki/InstallationSteps.twiki dca0618e3 
  graphdb/janus/pom.xml 016a09c33 
  pom.xml 4d2ab7391 
  webapp/pom.xml 284f538f7 

Diff: https://reviews.apache.org/r/66064/diff/2/

Changes: https://reviews.apache.org/r/66064/diff/1-2/


We currently use this fix with our Atlas setup in a fork of the Atlas code, and 
have found no issues with it. Additionally, with the update to the 
berkley-elasticsearch profile, I have extensively tested that profile to make 
sure that management of Elasticsearch functioned correctly.


Pierre Padovani

Reply via email to