[FLINK-4739] [elasticsearch connector] Adding packaging details for the Elasticsearch connector
When an uber-jar containing an Elasticsearch sink is executed, an IllegalArgumentException may occur, which is caused by conflicting files of Elasticsearch and it's dependencies in META-INF/services. This commit adds further clarification to the documentation on how to build a porper uber-jar that can be properly executed by adapting the pom file. This closes #2591 Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/beb31fc1 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/beb31fc1 Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/beb31fc1 Branch: refs/heads/master Commit: beb31fc1c5a6f5db060e5a4e1d64eacb70632900 Parents: 8000b64 Author: Steffen Hausmann <[email protected]> Authored: Mon Oct 3 22:32:01 2016 +0200 Committer: Stephan Ewen <[email protected]> Committed: Wed Oct 5 19:36:14 2016 +0200 ---------------------------------------------------------------------- docs/dev/connectors/elasticsearch2.md | 32 ++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/flink/blob/beb31fc1/docs/dev/connectors/elasticsearch2.md ---------------------------------------------------------------------- diff --git a/docs/dev/connectors/elasticsearch2.md b/docs/dev/connectors/elasticsearch2.md index 8eed690..5f4267e 100644 --- a/docs/dev/connectors/elasticsearch2.md +++ b/docs/dev/connectors/elasticsearch2.md @@ -139,3 +139,35 @@ This now provides a list of Elasticsearch Nodes to which the sink should connect via a `TransportClient`. More information about Elasticsearch can be found [here](https://elastic.co). + + +#### Packaging the Elasticsearch Connector into an Uber-jar + +For the execution of your Flink program, +it is recommended to build a so-called uber-jar (executable jar) containing all your dependencies +(see [here]({{site.baseurl}}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution) for further information). + +However, +when an uber-jar containing an Elasticsearch sink is executed, +an `IllegalArgumentException` may occur, +which is caused by conflicting files of Elasticsearch and it's dependencies +in `META-INF/services`: + +``` +IllegalArgumentException[An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'Lucene50' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [es090, completion090, XBloomFilter]] +``` + +If the uber-jar is build by means of maven, +this issue can be avoided by adding the following bits to the pom file: + +``` +<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> + <resource>META-INF/services/org.apache.lucene.codecs.Codec</resource> +</transformer> +<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> + <resource>META-INF/services/org.apache.lucene.codecs.DocValuesFormat</resource> +</transformer> +<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> + <resource>META-INF/services/org.apache.lucene.codecs.PostingsFormat</resource> +</transformer> +```
