egalpin opened a new pull request #14347:
URL: https://github.com/apache/beam/pull/14347
This change set represents a rather large (and backward compatible) change
to the way ElastichsearchIO.Write operates. Presently, the Write transform has
2 responsibilities:
1. Convert input documents into Bulk API entities, serializing based on user
settings (partial update, delete, upsert, etc) -> `DocToBulk`
2. Batch the converted Bulk API entities together and interface with the
target ES cluster -> `BulkIO`
This PR aims to separate these 2 responsibilities into discrete PTransforms
to allow for greater flexibility while also maintaining the convenience of the
Write transform to perform both document conversion and IO serially. Examples
of how the flexibility of separate transforms could be used:
1. Unit testing. It becomes trivial for pipeline developers to ensure that
output Bulk API entities for a given set of inputs will produce an expected
result, without the need for an available Elasticsearch cluster.
2. Flexible options for data backup. Serialized Bulk API entities can be
forked and sent to both Elasticsearch and a data lake.
3. Mirroring data to multiple clusters. Presently, mirroring data to
multiple clusters would require duplicate computation.
4. Better batching with input streams in one job. A job may produce multiple
"shapes" of Bulk API entities based on multiple input types, and then "fan-in"
all serialized Bulk entities into a single BulkIO transform to improve batching
semantics.
5. Decoupled jobs. Corollary to (3) above. Job(s) could be made to produce
Bulk entities and then publish them to a message bus. A distinct job could
consume from that message bus and solely be responsible for IO with the target
cluster(s).
6. Easier support for multiple BulkIO semantics.
Expanding on point (6), this PR also introduces a new (optional) way to
batch entities for bulk requests: Stateful Processing. Presently, Bulk request
size is limited by the lesser of Runner bundle size and `maxBatchSize` user
setting. In my experience, bundle sizes are often very small, and can be a
small as 1 or 2. When that’s the case, it means Bulk requests contain only 1 or
2 documents, and it’s effectively the same as not using the Bulk API at all.
`BulkIOStatefulFn` is made to be compatible with `GroupIntoBatches` which will
use entity count and (optionally) elapsed time to create batches much closer to
the `maxBatchSize` setting to improve throughput.
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] [**Choose
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>SDK</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/";>
<img
src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>---</td>
<td>---</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>---</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon";>
</a>
<a
href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon";>
</a><br>
</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon";>
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>---</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>---</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>---</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>---</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon";>
</a><br>
</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/";>
<img
src="https://camo.githubusercontent.com/4565d7b7e907114e6c1d12323408bd903aa252fefed5eeab93701b05c9628a84/68747470733a2f2f63692d6265616d2e6170616368652e6f72672f6a6f622f6265616d5f507265436f6d6d69745f507974686f6e446f636b65725f43726f6e2f62616467652f69636f6e";
alt="Build Status"
data-canonical-src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon";>
</a><br>
<a
href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/";>
<img
src="https://camo.githubusercontent.com/21afb097a5745440598bee5c59a027b140585eec871c3f3b883200610fabf722/68747470733a2f2f63692d6265616d2e6170616368652e6f72672f6a6f622f6265616d5f507265436f6d6d69745f507974686f6e446f63735f43726f6e2f62616467652f69636f6e";
alt="Build Status"
data-canonical-src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon";>
</a>
</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon";>
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a
href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/";>
<img alt="Build Status"
src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon";>
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See
[.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md)
for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
