-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/
-----------------------------------------------------------
(Updated April 13, 2018, 2:55 a.m.)
Review request for atlas, David Radley, Madhan Neethiraj, and Sarath
Subramanian.
Changes
-------
This update is a rebase against master. It also resolves an inconsistent
startup of elasticsearch.
This partially resolves some of the issues found in this review, but I believe
further discussion is needed to resolve things correctly.
Issues:
- Berkley db is excluded from the distribution, so there is this copy hack
that must happen. This has always been the case and falls outside the scope of
this patch
- The way I configure the Constants is not ideal. I am in need of guidance on
where/how to implement this fix as my understanding of the internals of Atlas
is limited.
- The connection to elasticsearch is now done by REST and not by native
client. The jars do not contain any shaded 3rd party jars, and are needed for
the REST client. If we build/distribute a full distribution that isn't based on
a particular profile, these jars will be required if someone wants to use
Elasticsearch. They should be harmless due to the lack of shaded 3rd party jars.
- I need to add some documentation around using this profile, but wish to
hold off until we have agreement on what and how to deal with the Elasticsearch
jars.
To Run this profile:
mvn clean install -Pdist,berkeley-elasticsearch -DskipTests
cd distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT
./bin/atlas_start.py
./bin/atlas_stop.py <-- Atlas will NOT start correctly becuase the berkley db
jar is missing, starting and stopping will create the lib dir where the berkley
db jar is needed.
cp
../../../../webapp/target/atlas-webapp-1.0.0-SNAPSHOT/WEB-INF/lib/je-7.3.7.jar
server/webapp/atlas/WEB-INF/lib
./bin/atlas_start.py
Bugs: ATLAS-2478
https://issues.apache.org/jira/browse/ATLAS-2478
Repository: atlas
Description
-------
This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates
documentation.
Included with this patch is an update to the berkley-elasticsearch profile to
automatically download and include elasticsearch as a side application much
like solr is. Updates to the start/stop/conf scripts are included as well.
NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to
/atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There
are six constants that are incorrectly named with a '.' (dot). This is not
supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the
field names can be collectively thought of as an object. In the case of the
fields defined in the Constants.java file, 'type' is defined as a string field,
and 'type.name' is also defined as a string field. Elasticsearch sees this as
an error, since it cannot convert type to an object. The fix included simply
changes the field names from using a '.' (dot) to an '_' (underscore). This
should NOT affect compatibility with hbase/solr for new installs. For existing
installations, a reindex will be required as the field names will have changed.
**Query**: There is a way we can simplify integration/unit tests for the
in-memory graph store by using a maven plugin that will download and run an
elasticsearch node. This is nothing more than a maven change, and change to the
atlas-application.properties to switch to elasticsearch from solr. I did not
implement this, but am curious if this change would be desired. If so, this can
be done with a separate ticket.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/repository/Constants.java 01e49157d
distro/pom.xml 1f4c6d557
distro/src/bin/atlas_config.py 9062da649
distro/src/bin/atlas_start.py 61d69eb21
distro/src/bin/atlas_stop.py 94c3d6d46
distro/src/conf/atlas-env.sh 9213f488b
distro/src/main/assemblies/standalone-package.xml dc2a66ba9
docs/src/site/twiki/Configuration.twiki 63c3fce96
docs/src/site/twiki/HighAvailability.twiki 4270d0974
docs/src/site/twiki/InstallationSteps.twiki dca0618e3
graphdb/janus/pom.xml 016a09c33
pom.xml 4d2ab7391
webapp/pom.xml 284f538f7
Diff: https://reviews.apache.org/r/66064/diff/2/
Changes: https://reviews.apache.org/r/66064/diff/1-2/
Testing
-------
We currently use this fix with our Atlas setup in a fork of the Atlas code, and
have found no issues with it. Additionally, with the update to the
berkley-elasticsearch profile, I have extensively tested that profile to make
sure that management of Elasticsearch functioned correctly.
Thanks,
Pierre Padovani
