This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 24752a2 Publishing website 2019/08/19 12:24:49 at commit 0467ac4
24752a2 is described below
commit 24752a2954f0af41c6521f45b6e8fb0882a558c8
Author: jenkins <[email protected]>
AuthorDate: Mon Aug 19 12:24:49 2019 +0000
Publishing website 2019/08/19 12:24:49 at commit 0467ac4
---
.../documentation/io/built-in/index.html | 4 +-
.../io/built-in/{ => parquet}/index.html | 253 ++++++++++-----------
2 files changed, 128 insertions(+), 129 deletions(-)
diff --git a/website/generated-content/documentation/io/built-in/index.html
b/website/generated-content/documentation/io/built-in/index.html
index 431a3fe..766f0c8 100644
--- a/website/generated-content/documentation/io/built-in/index.html
+++ b/website/generated-content/documentation/io/built-in/index.html
@@ -478,7 +478,7 @@ limitations under the License.
<p><a
href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java";>TFRecordIO</a></p>
<p><a
href="https://github.com/apache/beam/blob/master/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/XmlIO.java";>XmlIO</a></p>
<p><a
href="https://github.com/apache/beam/blob/master/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/TikaIO.java";>TikaIO</a></p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java";>ParquetIO</a></p>
+ <p><a href="/documentation/io/built-in/parquet">ParquetIO</a></p>
</td>
<td>
<p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/kinesis";>Amazon
Kinesis</a></p>
@@ -512,7 +512,7 @@ limitations under the License.
<td>
<p>Beam Python supports Apache HDFS, Google Cloud Storage, and local
filesystems.</p>
<p><a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/avroio.py";>avroio</a></p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/parquetio.py";>parquetio</a></p>
+ <p><a href="/documentation/io/built-in/parquet">parquetio.py</a></p>
<p><a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/textio.py";>textio</a></p>
<p><a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/tfrecordio.py";>tfrecordio</a></p>
<p><a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/vcfio.py";>vcfio</a></p>
diff --git a/website/generated-content/documentation/io/built-in/index.html
b/website/generated-content/documentation/io/built-in/parquet/index.html
similarity index 67%
copy from website/generated-content/documentation/io/built-in/index.html
copy to website/generated-content/documentation/io/built-in/parquet/index.html
index 431a3fe..cb60328 100644
--- a/website/generated-content/documentation/io/built-in/index.html
+++ b/website/generated-content/documentation/io/built-in/parquet/index.html
@@ -28,7 +28,7 @@
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Built-in I/O Transforms</title>
+ <title>Apache Parquet I/O connector</title>
<meta name="description" content="Apache Beam is an open source, unified
model and set of language-specific SDKs for defining and executing data
processing workflows, and also data ingestion and integration flows, supporting
Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs).
Dataflow pipelines simplify the mechanics of large-scale batch and streaming
data processing and can run on a number of runtimes like Apache Flink, Apache
Spark, and Google Cloud Dataflow [...]
">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400";
rel="stylesheet">
@@ -39,7 +39,7 @@
<script src="/js/fix-menu.js"></script>
<script src="/js/section-nav.js"></script>
<script src="/js/page-nav.js"></script>
- <link rel="canonical"
href="https://beam.apache.org/documentation/io/built-in/"; data-proofer-ignore>
+ <link rel="canonical"
href="https://beam.apache.org/documentation/io/built-in/parquet/";
data-proofer-ignore>
<link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
<link rel="alternate" type="application/rss+xml" title="Apache Beam"
href="https://beam.apache.org/feed.xml";>
<link rel="stylesheet"
href="https://use.fontawesome.com/releases/v5.4.1/css/all.css";
integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz"
crossorigin="anonymous">
@@ -141,7 +141,7 @@
GitHub links will not resolve until the markdown source is
available on the master branch.
New pages would fail validation during development / PR test
automation.
-->
- <a
href="https://github.com/apache/beam/edit/master/website/src/documentation/io/built-in.md";
data-proofer-ignore>
+ <a
href="https://github.com/apache/beam/edit/master/website/src/documentation/io/built-in-parquet.md";
data-proofer-ignore>
<i class="far fa-edit fa-lg" alt="Edit on GitHub" title="Edit on
GitHub"></i>
</a>
</li>
@@ -436,6 +436,9 @@
+<ul class="nav">
+ <li><a href="#before-you-start">Before you start</a></li>
+</ul>
</nav>
@@ -455,130 +458,126 @@ See the License for the specific language governing
permissions and
limitations under the License.
-->
-<h1 id="built-in-io-transforms">Built-in I/O Transforms</h1>
-
-<p>This table contains the currently available I/O transforms.</p>
-
-<p>Consult the <a
href="/documentation/programming-guide#pipeline-io">Programming Guide I/O
section</a> for general usage instructions, and see the javadoc/pydoc for the
particular I/O transforms.</p>
-
-<table class="table table-bordered">
-<tr>
- <th>Language</th>
- <th>File-based</th>
- <th>Messaging</th>
- <th>Database</th>
-</tr>
-<tr>
- <td>Java</td>
- <td>
- <p>Beam Java supports Apache HDFS, Amazon S3, Google Cloud Storage, and
local filesystems.</p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java";>FileIO</a>
(general-purpose reading, writing, and matching of files)</p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java";>AvroIO</a></p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java";>TextIO</a></p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java";>TFRecordIO</a></p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/XmlIO.java";>XmlIO</a></p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/TikaIO.java";>TikaIO</a></p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java";>ParquetIO</a></p>
- </td>
- <td>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/kinesis";>Amazon
Kinesis</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/amqp";>AMQP</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/kafka";>Apache
Kafka</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub";>Google
Cloud Pub/Sub</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/jms";>JMS</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/mqtt";>MQTT</a></p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java";>RabbitMqIO</a></p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsIO.java";>SqsIO</a></p>
- </td>
- <td>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/cassandra";>Apache
Cassandra</a></p>
- <p><a href="/documentation/io/built-in/hadoop/">Apache Hadoop Input/Output
Format</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/hbase";>Apache
HBase</a></p>
- <p><a href="/documentation/io/built-in/hcatalog">Apache Hive
(HCatalog)</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/kudu";>Apache
Kudu</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/solr";>Apache
Solr</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/elasticsearch";>Elasticsearch
(v2.x, v5.x, v6.x)</a></p>
- <p><a href="/documentation/io/built-in/google-bigquery/">Google
BigQuery</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable";>Google
Cloud Bigtable</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore";>Google
Cloud Datastore</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner";>Google
Cloud Spanner</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/jdbc";>JDBC</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/mongodb";>MongoDB</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/java/io/redis";>Redis</a></p>
- </td>
-</tr>
-<tr>
- <td>Python/Batch</td>
- <td>
- <p>Beam Python supports Apache HDFS, Google Cloud Storage, and local
filesystems.</p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/avroio.py";>avroio</a></p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/parquetio.py";>parquetio</a></p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/textio.py";>textio</a></p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/tfrecordio.py";>tfrecordio</a></p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/vcfio.py";>vcfio</a></p>
- </td>
- <td>
- </td>
- <td>
- <p><a href="/documentation/io/built-in/google-bigquery/">Google
BigQuery</a></p>
- <p><a
href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/io/gcp/datastore";>Google
Cloud Datastore</a></p>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigtableio.py";>Google
Cloud Bigtable</a> (Write)</p>
- </td>
-</tr>
-<tr>
- <td>Python/Streaming</td>
- <td>
- </td>
- <td>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/pubsub.py";>Google
Cloud Pub/Sub</a></p>
- </td>
- <td>
- <p><a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py";>Google
BigQuery (sink only)</a></p>
- </td>
-</tr>
-</table>
-
-<h1 id="in-progress-io-transforms">In-Progress I/O Transforms</h1>
-
-<p>This table contains I/O transforms that are currently planned or
in-progress. Status information can be found on the JIRA issue, or on the
GitHub PR linked to by the JIRA issue (if there is one).</p>
-
-<table class="table table-bordered">
- <tr>
- <th>Name</th><th>Language</th><th>JIRA</th>
- </tr>
- <tr>
- <td>Apache DistributedLog</td><td>Java</td>
- <td><a
href="https://issues.apache.org/jira/browse/BEAM-607";>BEAM-607</a></td>
- </tr>
- <tr>
- <td>Apache Kafka</td><td>Python</td>
- <td><a
href="https://issues.apache.org/jira/browse/BEAM-3788";>BEAM-3788</a></td>
- </tr>
- <tr>
- <td>Apache Sqoop</td><td>Java</td>
- <td><a
href="https://issues.apache.org/jira/browse/BEAM-67";>BEAM-67</a></td>
- </tr>
- <tr>
- <td>Couchbase</td><td>Java</td>
- <td><a
href="https://issues.apache.org/jira/browse/BEAM-1893";>BEAM-1893</a></td>
- </tr>
- <tr>
- <td>InfluxDB</td><td>Java</td>
- <td><a
href="https://issues.apache.org/jira/browse/BEAM-2546";>BEAM-2546</a></td>
- </tr>
- <tr>
- <td>Memcached</td><td>Java</td>
- <td><a
href="https://issues.apache.org/jira/browse/BEAM-1678";>BEAM-1678</a></td>
- </tr>
- <tr>
- <td>Neo4j</td><td>Java</td>
- <td><a
href="https://issues.apache.org/jira/browse/BEAM-1857";>BEAM-1857</a></td>
- </tr>
- <tr>
- <td>RestIO</td><td>Java</td>
- <td><a
href="https://issues.apache.org/jira/browse/BEAM-1946";>BEAM-1946</a></td>
- </tr>
-</table>
+<p><a href="/documentation/io/built-in/">Built-in I/O Transforms</a></p>
+
+<h1 id="apache-parquet-io-connector">Apache Parquet I/O connector</h1>
+
+<nav class="language-switcher">
+ <strong>Adapt for:</strong>
+ <ul>
+ <li data-type="language-java" class="active">Java SDK</li>
+ <li data-type="language-py">Python SDK</li>
+ </ul>
+</nav>
+
+<p>The Beam SDKs include built-in transforms that can read data from and write
data
+to <a href="https://parquet.apache.org";>Apache Parquet</a> files.</p>
+
+<h2 id="before-you-start">Before you start</h2>
+
+<!-- Java specific -->
+
+<p class="language-java">To use ParquetIO, add the Maven artifact dependency
to your <code class="highlighter-rouge">pom.xml</code> file.</p>
+
+<div class="language-java highlighter-rouge"><pre
class="highlight"><code><span class="o"><</span><span
class="n">dependency</span><span class="o">></span>
+ <span class="o"><</span><span class="n">groupId</span><span
class="o">></span><span class="n">org</span><span class="o">.</span><span
class="na">apache</span><span class="o">.</span><span
class="na">beam</span><span class="o"></</span><span
class="n">groupId</span><span class="o">></span>
+ <span class="o"><</span><span class="n">artifactId</span><span
class="o">></span><span class="n">beam</span><span class="o">-</span><span
class="n">sdks</span><span class="o">-</span><span class="n">java</span><span
class="o">-</span><span class="n">io</span><span class="o">-</span><span
class="n">parquet</span><span class="o"></</span><span
class="n">artifactId</span><span class="o">></span>
+ <span class="o"><</span><span class="n">version</span><span
class="o">></span><span class="mf">2.14</span><span class="o">.</span><span
class="mi">0</span><span class="o"></</span><span
class="n">version</span><span class="o">></span>
+<span class="o"></</span><span class="n">dependency</span><span
class="o">></span>
+</code></pre>
+</div>
+
+<p class="language-java">Additional resources:</p>
+
+<ul class="language-java">
+ <li><a
href="https://github.com/apache/beam/blob/master/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java";>ParquetIO
source code</a></li>
+ <li><a
href="https://beam.apache.org/releases/javadoc/2.14.0/org/apache/beam/sdk/io/parquet/ParquetIO.html";>ParquetIO
Javadoc</a></li>
+</ul>
+
+<!-- Python specific -->
+
+<p class="language-py">ParquetIO comes preinstalled with the Apache Beam
python sdk..</p>
+
+<p class="language-py">Additional resources:</p>
+
+<ul class="language-py">
+ <li><a
href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/parquetio.py";>ParquetIO
source code</a></li>
+ <li><a
href="https://beam.apache.org/releases/pydoc/2.14.0/apache_beam.io.parquetio.html";>ParquetIO
Pydoc</a></li>
+</ul>
+
+<h4 class="language-java" id="using-parquetio-with-spark-before-24">Using
ParquetIO with Spark before 2.4</h4>
+
+<p class="language-java"><code class="highlighter-rouge">ParquetIO</code>
depends on an API introduced in Apache Parquet 1.10.0. <strong>Spark 2.4.x is
compatible and no additional steps are necessary</strong>. Older versions of
Spark will not work out of the box since a pre-installed version of Parquet
libraries will take precedence during execution. The following workaround
should be applied.</p>
+
+<blockquote class="language-java">
+ <p><strong>Note</strong>: The following technique allows you to execute your
pipeline with <code class="highlighter-rouge">ParquetIO</code> correctly.
+The Parquet files that are consumed or generated by this Beam connector should
remain interoperable with the other tools on your cluster.</p>
+</blockquote>
+
+<p class="language-java">Include the Parquet artifact normally and ensure that
it brings in the correct version of Parquet as a transitive dependency.</p>
+
+<div class="language-java highlighter-rouge"><pre
class="highlight"><code><span class="o"><</span><span
class="n">dependency</span><span class="o">></span>
+ <span class="o"><</span><span class="n">groupId</span><span
class="o">></span><span class="n">org</span><span class="o">.</span><span
class="na">apache</span><span class="o">.</span><span
class="na">beam</span><span class="o"></</span><span
class="n">groupId</span><span class="o">></span>
+ <span class="o"><</span><span class="n">artifactId</span><span
class="o">></span><span class="n">beam</span><span class="o">-</span><span
class="n">sdks</span><span class="o">-</span><span class="n">java</span><span
class="o">-</span><span class="n">io</span><span class="o">-</span><span
class="n">parquet</span><span class="o"></</span><span
class="n">artifactId</span><span class="o">></span>
+ <span class="o"><</span><span class="n">version</span><span
class="o">></span><span class="err">$</span><span class="o">{</span><span
class="n">beam</span><span class="o">.</span><span
class="na">version</span><span class="o">}</</span><span
class="n">version</span><span class="o">></span>
+<span class="o"></</span><span class="n">dependency</span><span
class="o">></span>
+</code></pre>
+</div>
+
+<p class="language-java">Relocate the following packages:</p>
+
+<div class="language-java highlighter-rouge"><pre
class="highlight"><code><span class="o"><</span><span
class="n">plugin</span><span class="o">></span>
+ <span class="o"><</span><span class="n">groupId</span><span
class="o">></span><span class="n">org</span><span class="o">.</span><span
class="na">apache</span><span class="o">.</span><span
class="na">maven</span><span class="o">.</span><span
class="na">plugins</span><span class="o"></</span><span
class="n">groupId</span><span class="o">></span>
+ <span class="o"><</span><span class="n">artifactId</span><span
class="o">></span><span class="n">maven</span><span class="o">-</span><span
class="n">shade</span><span class="o">-</span><span
class="n">plugin</span><span class="o"></</span><span
class="n">artifactId</span><span class="o">></span>
+ <span class="o"><</span><span class="n">configuration</span><span
class="o">></span>
+ <span class="o"><</span><span
class="n">createDependencyReducedPom</span><span class="o">></span><span
class="kc">false</span><span class="o"></</span><span
class="n">createDependencyReducedPom</span><span class="o">></span>
+ <span class="o"><</span><span class="n">filters</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">filter</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">artifact</span><span
class="o">>*:*</</span><span class="n">artifact</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">excludes</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">exclude</span><span
class="o">></span><span class="n">META</span><span class="o">-</span><span
class="n">INF</span><span class="o">/*.</span><span class="na">SF</span><span
class="o"></</span><span class="n">exclude</span><span class="o">></span>
+ <span class="o"><</span><span class="n">exclude</span><span
class="o">></span><span class="n">META</span><span class="o">-</span><span
class="n">INF</span><span class="o">/*.</span><span class="na">DSA</span><span
class="o"></</span><span class="n">exclude</span><span class="o">></span>
+ <span class="o"><</span><span class="n">exclude</span><span
class="o">></span><span class="n">META</span><span class="o">-</span><span
class="n">INF</span><span class="o">/*.</span><span class="na">RSA</span><span
class="o"></</span><span class="n">exclude</span><span class="o">></span>
+ <span class="o"></</span><span class="n">excludes</span><span
class="o">></span>
+ <span class="o"></</span><span class="n">filter</span><span
class="o">></span>
+ <span class="o"></</span><span class="n">filters</span><span
class="o">></span>
+ <span class="o"></</span><span class="n">configuration</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">executions</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">execution</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">phase</span><span
class="o">></span><span class="kn">package</span><span
class="o"></</span><span class="n">phase</span><span class="o">></span>
+ <span class="o"><</span><span class="n">goals</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">goal</span><span
class="o">></span><span class="n">shade</span><span
class="o"></</span><span class="n">goal</span><span class="o">></span>
+ <span class="o"></</span><span class="n">goals</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">configuration</span><span
class="o">></span>
+ <span class="o"><</span><span
class="n">shadedArtifactAttached</span><span class="o">></span><span
class="kc">true</span><span class="o"></</span><span
class="n">shadedArtifactAttached</span><span class="o">></span>
+ <span class="o"><</span><span
class="n">shadedClassifierName</span><span class="o">></span><span
class="n">shaded</span><span class="o"></</span><span
class="n">shadedClassifierName</span><span class="o">></span>
+ <span class="o"><</span><span class="n">relocations</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">relocation</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">pattern</span><span
class="o">></span><span class="n">org</span><span class="o">.</span><span
class="na">apache</span><span class="o">.</span><span
class="na">parquet</span><span class="o"></</span><span
class="n">pattern</span><span class="o">></span>
+ <span class="o"><</span><span
class="n">shadedPattern</span><span class="o">></span><span
class="n">shaded</span><span class="o">.</span><span class="na">org</span><span
class="o">.</span><span class="na">apache</span><span class="o">.</span><span
class="na">parquet</span><span class="o"></</span><span
class="n">shadedPattern</span><span class="o">></span>
+ <span class="o"></</span><span class="n">relocation</span><span
class="o">></span>
+ <span class="o"><!--</span> <span class="n">Some</span> <span
class="n">packages</span> <span class="n">are</span> <span
class="n">shaded</span> <span class="n">already</span><span class="o">,</span>
<span class="n">and</span> <span class="n">on</span> <span class="n">the</span>
<span class="n">original</span> <span class="n">spark</span> <span
class="n">classpath</span><span class="o">.</span> <span class="n">Shade</span>
<span class="n">them</span> <span class="n">more</sp [...]
+ <span class="o"><</span><span class="n">relocation</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">pattern</span><span
class="o">></span><span class="n">shaded</span><span class="o">.</span><span
class="na">parquet</span><span class="o"></</span><span
class="n">pattern</span><span class="o">></span>
+ <span class="o"><</span><span
class="n">shadedPattern</span><span class="o">></span><span
class="n">reshaded</span><span class="o">.</span><span
class="na">parquet</span><span class="o"></</span><span
class="n">shadedPattern</span><span class="o">></span>
+ <span class="o"></</span><span class="n">relocation</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">relocation</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">pattern</span><span
class="o">></span><span class="n">org</span><span class="o">.</span><span
class="na">apache</span><span class="o">.</span><span
class="na">avro</span><span class="o"></</span><span
class="n">pattern</span><span class="o">></span>
+ <span class="o"><</span><span
class="n">shadedPattern</span><span class="o">></span><span
class="n">shaded</span><span class="o">.</span><span class="na">org</span><span
class="o">.</span><span class="na">apache</span><span class="o">.</span><span
class="na">avro</span><span class="o"></</span><span
class="n">shadedPattern</span><span class="o">></span>
+ <span class="o"></</span><span class="n">relocation</span><span
class="o">></span>
+ <span class="o"></</span><span class="n">relocations</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">transformers</span><span
class="o">></span>
+ <span class="o"><</span><span class="n">transformer</span>
+ <span class="n">implementation</span><span class="o">=</span><span
class="s">"org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"</span><span
class="o">/></span>
+ <span class="o"></</span><span class="n">transformers</span><span
class="o">></span>
+ <span class="o"></</span><span class="n">configuration</span><span
class="o">></span>
+ <span class="o"></</span><span class="n">execution</span><span
class="o">></span>
+ <span class="o"></</span><span class="n">executions</span><span
class="o">></span>
+<span class="o"></</span><span class="n">plugin</span><span
class="o">></span>
+</code></pre>
+</div>
+
+<p class="language-java">This technique has been tested to work on Spark
2.2.3, Spark 2.3.3 and Spark 2.4.3 (although it is optional for Spark 2.4+).</p>
</div>
</div>
