rszper commented on code in PR #25083: URL: https://github.com/apache/beam/pull/25083#discussion_r1124976782
########## website/www/site/content/en/documentation/ml/overview.md: ########## @@ -105,3 +105,4 @@ You can find examples of end-to-end AI/ML pipelines for several use cases: * [Anomaly Detection in Beam](/documentation/ml/anomaly-detection): Demonstrates how to set up an anomaly detection pipeline that reads text from Pub/Sub in real time and then detects anomalies using a trained HDBSCAN clustering model with the RunInference API. * [Large Language Model Inference in Beam](/documentation/ml/large-language-modeling): Demonstrates a pipeline that uses RunInference to perform translation with the T5 language model which contains 11 billion parameters. * [Per Entity Training in Beam](/documentation/ml/per-entity-training): Demonstrates a pipeline that trains a Decision Tree Classifier per education level for predicting if the salary of a person is >= 50k. +* [Cross-language RunInference in Beam](/documentation/ml/multi-language-inference): Demonstrates a cross-language java pipeline that uses a custom python transform to do preprocessing, postprocessing, and inference. Review Comment: Note that this page is being changed in https://github.com/apache/beam/pull/25700/files. If that PR is merged first, this could be added either to the Data processing or to the Prediction and inference examples, depending on which you think is more appropriate. ########## website/www/site/layouts/partials/section-menu/en/documentation.html: ########## @@ -1,367 +1,368 @@ {{/* - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. See accompanying LICENSE file. -*/}} - -<li><span class="section-nav-list-main-title">Documentation</span></li> -<li><a href="/documentation">Using the Documentation</a></li> -<li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Concepts</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/basics/">Basics of the Beam model</a></li> - <li><a href="/documentation/runtime/model/">How Beam executes a pipeline</a></li> - </ul> -</li> -<li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Beam programming guide</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/programming-guide/">Overview</a></li> - <li><a href="/documentation/programming-guide/#creating-a-pipeline">Pipelines</a></li> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">PCollections</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/programming-guide/#pcollections">Creating a PCollection</a></li> - <li><a href="/documentation/programming-guide/#pcollection-characteristics">PCollection characteristics</a></li> - </ul> - </li> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Transforms</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/programming-guide/#applying-transforms">Applying transforms</a></li> - <li> - <span class="section-nav-list-title">Core Beam transforms</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/programming-guide/#pardo">ParDo</a></li> - <li><a href="/documentation/programming-guide/#groupbykey">GroupByKey</a></li> - <li><a href="/documentation/programming-guide/#cogroupbykey">CoGroupByKey</a></li> - <li><a href="/documentation/programming-guide/#combine">Combine</a></li> - <li><a href="/documentation/programming-guide/#flatten">Flatten</a></li> - <li><a href="/documentation/programming-guide/#partition">Partition</a></li> - </ul> - </li> - - <li><a href="/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms">Requirements for user code</a></li> - <li><a href="/documentation/programming-guide/#side-inputs">Side inputs</a></li> - <li><a href="/documentation/programming-guide/#additional-outputs">Additional outputs</a></li> - <li><a href="/documentation/programming-guide/#composite-transforms">Composite transforms</a></li> - </ul> - </li> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Pipeline I/O</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li> - <li><a href="/documentation/io/connectors/">I/O connectors</a></li> - - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">I/O connector guides</span> - <ul class="section-nav-list"> - <li><a href="/documentation/io/built-in/parquet/">Apache Parquet I/O connector</a></li> - <li><a href="/documentation/io/built-in/hadoop/">Hadoop Input/Output Format IO</a></li> - <li><a href="/documentation/io/built-in/hcatalog/">HCatalog IO</a></li> - <li><a href="/documentation/io/built-in/google-bigquery/">Google BigQuery I/O connector</a></li> - <li><a href="/documentation/io/built-in/snowflake/">Snowflake I/O connector</a></li> - <li><a href="/documentation/io/built-in/cdap/">CDAP I/O connector</a></li> - <li><a href="/documentation/io/built-in/sparkreceiver/">Spark Receiver I/O connector</a></li> - <li><a href="/documentation/io/built-in/singlestore/">SingleStoreDB I/O connector</a></li> - </ul> - </li> - - - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Developing new I/O connectors</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li> - <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li> - <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li> - <li><a href="/documentation/io/io-standards/">I/O Standards</a></li> - </ul> - </li> - - <li><a href="/documentation/io/testing/">Testing I/O transforms</a></li> - </ul> - </li> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Schemas</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/programming-guide/#what-is-a-schema">What is a schema</a></li> - <li><a href="/documentation/programming-guide/#schemas-for-pl-types">Schemas for programming language types</a></li> - <li><a href="/documentation/programming-guide/#schema-definition">Schema definition</a></li> - <li><a href="/documentation/programming-guide/#logical-types">Logical types</a></li> - <li><a href="/documentation/programming-guide/#creating-schemas">Creating schemas</a></li> - <li><a href="/documentation/programming-guide/#using-schemas">Using schemas</a></li> - </ul> - </li> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Data encoding and type safety</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/programming-guide/#data-encoding-and-type-safety">Data encoding basics</a></li> - <li><a href="/documentation/programming-guide/#specifying-coders">Specifying coders</a></li> - <li><a href="/documentation/programming-guide/#default-coders-and-the-coderregistry">Default coders and the CoderRegistry</a></li> - </ul> - </li> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Windowing</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/programming-guide/#windowing">Windowing basics</a></li> - <li><a href="/documentation/programming-guide/#provided-windowing-functions">Provided windowing functions</a></li> - <li><a href="/documentation/programming-guide/#setting-your-pcollections-windowing-function">Setting your PCollection’s windowing function</a></li> - <li><a href="/documentation/programming-guide/#watermarks-and-late-data">Watermarks and late data</a></li> - <li><a href="/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements">Adding timestamps to a PCollection’s elements</a></li> - </ul> - </li> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Triggers</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/programming-guide/#triggers">Trigger basics</a></li> - <li><a href="/documentation/programming-guide/#event-time-triggers">Event time triggers and the default trigger</a></li> - <li><a href="/documentation/programming-guide/#processing-time-triggers">Processing time triggers</a></li> - <li><a href="/documentation/programming-guide/#data-driven-triggers">Data-driven triggers</a></li> - <li><a href="/documentation/programming-guide/#setting-a-trigger">Setting a trigger</a></li> - <li><a href="/documentation/programming-guide/#composite-triggers">Composite triggers</a></li> - </ul> - </li> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Metrics</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/programming-guide/#metrics">Metrics basics</a></li> - <li><a href="/documentation/programming-guide/#types-of-metrics">Types of metrics</a></li> - <li><a href="/documentation/programming-guide/#querying-metrics">Querying metrics</a></li> - <li><a href="/documentation/programming-guide/#using-metrics">Using metrics in pipeline</a></li> - <li><a href="/documentation/programming-guide/#export-metrics">Export metrics</a></li> - </ul> - </li> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">State and Timers</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/programming-guide/#types-of-state">Types of state</a></li> - <li><a href="/documentation/programming-guide/#deferred-state-reads">Deferred state reads</a></li> - <li><a href="/documentation/programming-guide/#timers">Timers</a></li> - <li><a href="/documentation/programming-guide/#garbage-collecting-state">Garbage collecting state</a></li> - <li><a href="/documentation/programming-guide/#state-timers-examples">State and timers examples</a></li> - </ul> - </li> - - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Splittable DoFns</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/programming-guide/#sdf-basics">Basics</a></li> - <li><a href="/documentation/programming-guide/#sizing-and-progress">Sizing and progress</a></li> - <li><a href="/documentation/programming-guide/#user-initiated-checkpoint">User-initiated checkpoint</a></li> - <li><a href="/documentation/programming-guide/#runner-initiated-split">Runner initiated split</a></li> - <li><a href="/documentation/programming-guide/#watermark-estimation">Watermark estimation</a></li> - <li><a href="/documentation/programming-guide/#truncating-during-drain">Truncating during drain</a></li> - <li><a href="/documentation/programming-guide/#bundle-finalization">Bundle finalization</a></li> - </ul> - </li> - - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Multi-language Pipelines</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/programming-guide/#create-x-lang-transforms">Creating cross-language transforms</a></li> - <li><a href="/documentation/programming-guide/#use-x-lang-transforms">Using cross-language transforms</a></li> - <li><a href="/documentation/programming-guide/#x-lang-transform-runner-support">Runner Support</a></li> - </ul> - </li> - <li><a href="/documentation/programming-guide/#batched-dofns">Batched DoFns</a></li> - </ul> -</li> -<li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Pipeline development lifecycle</span> - <ul class="section-nav-list"> - <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li> - <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li> - <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li> - </ul> -</li> -<li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Common pipeline patterns</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/patterns/overview/">Overview</a></li> - <li><a href="/documentation/patterns/file-processing/">File processing</a></li> - <li><a href="/documentation/patterns/side-inputs/">Side inputs</a></li> - <li><a href="/documentation/patterns/pipeline-options/">Pipeline options</a></li> - <li><a href="/documentation/patterns/custom-io/">Custom I/O</a></li> - <li><a href="/documentation/patterns/custom-windows/">Custom windows</a></li> - <li><a href="/documentation/patterns/bigqueryio/">BigQueryIO</a></li> - <li><a href="/documentation/patterns/ai-platform/">AI Platform</a></li> - <li><a href="/documentation/patterns/schema/">Schema</a></li> - <li><a href="/documentation/patterns/bqml/">BigQuery ML</a></li> - <li><a href="/documentation/patterns/grouping-elements-for-efficient-external-service-calls/">Grouping elements for efficient external service calls</a></li> - </ul> -</li> -<li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">AI/ML pipelines</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/ml/overview/">Overview</a></li> - <li><a href="/documentation/ml/data-processing/">Data processing</a></li> - <li><a href="/documentation/ml/runinference-metrics/">RunInference Metrics</a></li> - <li><a href="/documentation/ml/tensorrt-runinference">TensorRT Inference</a></li> - <li><a href="/documentation/ml/model-evaluation/">Model evaluation</a></li> - <li><a href="/documentation/ml/orchestration/">Workflow Orchestration</a></li> - <li><a href="/documentation/ml/multi-model-pipelines/">Multi-model pipelines</a></li> - <li><a href="/documentation/ml/online-clustering/">Online Clustering</a></li> - <li><a href="/documentation/ml/anomaly-detection/">Anomaly Detection</a></li> - <li><a href="/documentation/ml/large-language-modeling">Large Language Model Inference in Beam</a></li> - <li><a href="/documentation/ml/per-entity-training">Per Entity Training in Beam</a></li> - </ul> -</li> -<li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Runtime systems</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/runtime/environments/">Container environments</a></li> - <li><a href="/documentation/runtime/resource-hints/">Resource hints</a></li> - <li><a href="/documentation/runtime/sdk-harness-config/">SDK Harness Configuration</a></li> - </ul> -</li> -<li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Transform catalog</span> - - <ul class="section-nav-list"> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Python</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/transforms/python/overview/">Overview</a></li> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Element-wise</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/transforms/python/elementwise/filter/">Filter</a></li> - <li><a href="/documentation/transforms/python/elementwise/flatmap/">FlatMap</a></li> - <li><a href="/documentation/transforms/python/elementwise/keys/">Keys</a></li> - <li><a href="/documentation/transforms/python/elementwise/kvswap/">KvSwap</a></li> - <li><a href="/documentation/transforms/python/elementwise/map/">Map</a></li> - <li><a href="/documentation/transforms/python/elementwise/pardo/">ParDo</a></li> - <li><a href="/documentation/transforms/python/elementwise/partition/">Partition</a></li> - <li><a href="/documentation/transforms/python/elementwise/regex/">Regex</a></li> - <li><a href="/documentation/transforms/python/elementwise/reify/">Reify</a></li> - <li><a href="/documentation/transforms/python/elementwise/runinference/">RunInference</a></li> - <li><a href="/documentation/transforms/python/elementwise/tostring/">ToString</a></li> - <li><a href="/documentation/transforms/python/elementwise/values/">Values</a></li> - <li><a href="/documentation/transforms/python/elementwise/withtimestamps/">WithTimestamps</a></li> - </ul> - </li> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Aggregation</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/transforms/python/aggregation/cogroupbykey/">CoGroupByKey</a></li> - <li><a href="/documentation/transforms/python/aggregation/combineglobally/">CombineGlobally</a></li> - <li><a href="/documentation/transforms/python/aggregation/combineperkey/">CombinePerKey</a></li> - <li><a href="/documentation/transforms/python/aggregation/combinevalues/">CombineValues</a></li> - <li><a href="/documentation/transforms/python/aggregation/count/">Count</a></li> - <li><a href="/documentation/transforms/python/aggregation/distinct/">Distinct</a></li> - <li><a href="/documentation/transforms/python/aggregation/groupbykey/">GroupByKey</a></li> - <li><a href="/documentation/transforms/python/aggregation/groupby/">GroupBy</a></li> - <li><a href="/documentation/transforms/python/aggregation/groupintobatches/">GroupIntoBatches</a></li> - <li><a href="/documentation/transforms/python/aggregation/latest/">Latest</a></li> - <li><a href="/documentation/transforms/python/aggregation/max/">Max</a></li> - <li><a href="/documentation/transforms/python/aggregation/min/">Min</a></li> - <li><a href="/documentation/transforms/python/aggregation/mean/">Mean</a></li> - <li><a href="/documentation/transforms/python/aggregation/sample/">Sample</a></li> - <li><a href="/documentation/transforms/python/aggregation/sum/">Sum</a></li> - <li><a href="/documentation/transforms/python/aggregation/top/">Top</a></li> - </ul> - </li> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Other</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/transforms/python/other/create/">Create</a></li> - <li><a href="/documentation/transforms/python/other/flatten/">Flatten</a></li> - <li><a href="/documentation/transforms/python/other/reshuffle/">Reshuffle</a></li> - <li><a href="/documentation/transforms/python/other/windowinto/">WindowInto</a></li> - </ul> - </li> - </ul> - </li> - - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Java</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/transforms/java/overview/">Overview</a></li> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Element-wise</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/transforms/java/elementwise/filter/">Filter</a></li> - <li><a href="/documentation/transforms/java/elementwise/flatmapelements/">FlatMapElements</a></li> - <li><a href="/documentation/transforms/java/elementwise/keys/">Keys</a></li> - <li><a href="/documentation/transforms/java/elementwise/kvswap/">KvSwap</a></li> - <li><a href="/documentation/transforms/java/elementwise/mapelements/">MapElements</a></li> - <li><a href="/documentation/transforms/java/elementwise/pardo/">ParDo</a></li> - <li><a href="/documentation/transforms/java/elementwise/partition/">Partition</a></li> - <li><a href="/documentation/transforms/java/elementwise/regex/">Regex</a></li> - <li><a href="/documentation/transforms/java/elementwise/reify/">Reify</a></li> - <li><a href="/documentation/transforms/java/elementwise/values/">Values</a></li> - <li><a href="/documentation/transforms/java/elementwise/tostring/">ToString</a></li> - <li><a href="/documentation/transforms/java/elementwise/withkeys/">WithKeys</a></li> - <li><a href="/documentation/transforms/java/elementwise/withtimestamps/">WithTimestamps</a></li> - </ul> - </li> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Aggregation</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/transforms/java/aggregation/approximatequantiles/">ApproximateQuantiles</a></li> - <li><a href="/documentation/transforms/java/aggregation/approximateunique/">ApproximateUnique</a></li> - <li><a href="/documentation/transforms/java/aggregation/cogroupbykey/">CoGroupByKey</a></li> - <li><a href="/documentation/transforms/java/aggregation/combine/">Combine</a></li> - <li><a href="/documentation/transforms/java/aggregation/combinewithcontext/">CombineWithContext</a></li> - <li><a href="/documentation/transforms/java/aggregation/count/">Count</a></li> - <li><a href="/documentation/transforms/java/aggregation/distinct/">Distinct</a></li> - <li><a href="/documentation/transforms/java/aggregation/groupbykey/">GroupByKey</a></li> - <li><a href="/documentation/transforms/java/aggregation/groupintobatches/">GroupIntoBatches</a></li> - <li><a href="/documentation/transforms/java/aggregation/hllcount/">HllCount</a></li> - <li><a href="/documentation/transforms/java/aggregation/latest/">Latest</a></li> - <li><a href="/documentation/transforms/java/aggregation/max/">Max</a></li> - <li><a href="/documentation/transforms/java/aggregation/mean/">Mean</a></li> - <li><a href="/documentation/transforms/java/aggregation/min/">Min</a></li> - <li><a href="/documentation/transforms/java/aggregation/sample/">Sample</a></li> - <li><a href="/documentation/transforms/java/aggregation/sum/">Sum</a></li> - <li><a href="/documentation/transforms/java/aggregation/top/">Top</a></li> - </ul> - </li> - <li class="section-nav-item--collapsible"> - <span class="section-nav-list-title">Other</span> - - <ul class="section-nav-list"> - <li><a href="/documentation/transforms/java/other/create/">Create</a></li> - <li><a href="/documentation/transforms/java/other/flatten/">Flatten</a></li> - <li><a href="/documentation/transforms/java/other/passert/">PAssert</a></li> - <li><a href="/documentation/transforms/java/other/view/">View</a></li> - <li><a href="/documentation/transforms/java/other/window/">Window</a></li> - </ul> - </li> - </ul> - </li> - </ul> -</li> -<li><a href="/documentation/glossary/">Glossary</a></li> -<li><a href="https://cwiki.apache.org/confluence/display/BEAM/Apache+Beam";>Beam Wiki <img src="/images/external-link-icon.png" width="14" height="14" alt="External link."></a></li> + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. + */}} + + <li><span class="section-nav-list-main-title">Documentation</span></li> + <li><a href="/documentation">Using the Documentation</a></li> + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">Concepts</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/basics/">Basics of the Beam model</a></li> + <li><a href="/documentation/runtime/model/">How Beam executes a pipeline</a></li> + </ul> + </li> + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">Beam programming guide</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/programming-guide/">Overview</a></li> + <li><a href="/documentation/programming-guide/#creating-a-pipeline">Pipelines</a></li> + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">PCollections</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/programming-guide/#pcollections">Creating a PCollection</a></li> + <li><a href="/documentation/programming-guide/#pcollection-characteristics">PCollection characteristics</a></li> + </ul> + </li> + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">Transforms</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/programming-guide/#applying-transforms">Applying transforms</a></li> + <li> + <span class="section-nav-list-title">Core Beam transforms</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/programming-guide/#pardo">ParDo</a></li> + <li><a href="/documentation/programming-guide/#groupbykey">GroupByKey</a></li> + <li><a href="/documentation/programming-guide/#cogroupbykey">CoGroupByKey</a></li> + <li><a href="/documentation/programming-guide/#combine">Combine</a></li> + <li><a href="/documentation/programming-guide/#flatten">Flatten</a></li> + <li><a href="/documentation/programming-guide/#partition">Partition</a></li> + </ul> + </li> + + <li><a href="/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms">Requirements for user code</a></li> + <li><a href="/documentation/programming-guide/#side-inputs">Side inputs</a></li> + <li><a href="/documentation/programming-guide/#additional-outputs">Additional outputs</a></li> + <li><a href="/documentation/programming-guide/#composite-transforms">Composite transforms</a></li> + </ul> + </li> + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">Pipeline I/O</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li> + <li><a href="/documentation/io/connectors/">I/O connectors</a></li> + + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">I/O connector guides</span> + <ul class="section-nav-list"> + <li><a href="/documentation/io/built-in/parquet/">Apache Parquet I/O connector</a></li> + <li><a href="/documentation/io/built-in/hadoop/">Hadoop Input/Output Format IO</a></li> + <li><a href="/documentation/io/built-in/hcatalog/">HCatalog IO</a></li> + <li><a href="/documentation/io/built-in/google-bigquery/">Google BigQuery I/O connector</a></li> + <li><a href="/documentation/io/built-in/snowflake/">Snowflake I/O connector</a></li> + <li><a href="/documentation/io/built-in/cdap/">CDAP I/O connector</a></li> + <li><a href="/documentation/io/built-in/sparkreceiver/">Spark Receiver I/O connector</a></li> + <li><a href="/documentation/io/built-in/singlestore/">SingleStoreDB I/O connector</a></li> + </ul> + </li> + + + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">Developing new I/O connectors</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li> + <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li> + <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li> + <li><a href="/documentation/io/io-standards/">I/O Standards</a></li> + </ul> + </li> + + <li><a href="/documentation/io/testing/">Testing I/O transforms</a></li> + </ul> + </li> + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">Schemas</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/programming-guide/#what-is-a-schema">What is a schema</a></li> + <li><a href="/documentation/programming-guide/#schemas-for-pl-types">Schemas for programming language types</a></li> + <li><a href="/documentation/programming-guide/#schema-definition">Schema definition</a></li> + <li><a href="/documentation/programming-guide/#logical-types">Logical types</a></li> + <li><a href="/documentation/programming-guide/#creating-schemas">Creating schemas</a></li> + <li><a href="/documentation/programming-guide/#using-schemas">Using schemas</a></li> + </ul> + </li> + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">Data encoding and type safety</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/programming-guide/#data-encoding-and-type-safety">Data encoding basics</a></li> + <li><a href="/documentation/programming-guide/#specifying-coders">Specifying coders</a></li> + <li><a href="/documentation/programming-guide/#default-coders-and-the-coderregistry">Default coders and the CoderRegistry</a></li> + </ul> + </li> + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">Windowing</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/programming-guide/#windowing">Windowing basics</a></li> + <li><a href="/documentation/programming-guide/#provided-windowing-functions">Provided windowing functions</a></li> + <li><a href="/documentation/programming-guide/#setting-your-pcollections-windowing-function">Setting your PCollection’s windowing function</a></li> + <li><a href="/documentation/programming-guide/#watermarks-and-late-data">Watermarks and late data</a></li> + <li><a href="/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements">Adding timestamps to a PCollection’s elements</a></li> + </ul> + </li> + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">Triggers</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/programming-guide/#triggers">Trigger basics</a></li> + <li><a href="/documentation/programming-guide/#event-time-triggers">Event time triggers and the default trigger</a></li> + <li><a href="/documentation/programming-guide/#processing-time-triggers">Processing time triggers</a></li> + <li><a href="/documentation/programming-guide/#data-driven-triggers">Data-driven triggers</a></li> + <li><a href="/documentation/programming-guide/#setting-a-trigger">Setting a trigger</a></li> + <li><a href="/documentation/programming-guide/#composite-triggers">Composite triggers</a></li> + </ul> + </li> + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">Metrics</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/programming-guide/#metrics">Metrics basics</a></li> + <li><a href="/documentation/programming-guide/#types-of-metrics">Types of metrics</a></li> + <li><a href="/documentation/programming-guide/#querying-metrics">Querying metrics</a></li> + <li><a href="/documentation/programming-guide/#using-metrics">Using metrics in pipeline</a></li> + <li><a href="/documentation/programming-guide/#export-metrics">Export metrics</a></li> + </ul> + </li> + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">State and Timers</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/programming-guide/#types-of-state">Types of state</a></li> + <li><a href="/documentation/programming-guide/#deferred-state-reads">Deferred state reads</a></li> + <li><a href="/documentation/programming-guide/#timers">Timers</a></li> + <li><a href="/documentation/programming-guide/#garbage-collecting-state">Garbage collecting state</a></li> + <li><a href="/documentation/programming-guide/#state-timers-examples">State and timers examples</a></li> + </ul> + </li> + + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">Splittable DoFns</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/programming-guide/#sdf-basics">Basics</a></li> + <li><a href="/documentation/programming-guide/#sizing-and-progress">Sizing and progress</a></li> + <li><a href="/documentation/programming-guide/#user-initiated-checkpoint">User-initiated checkpoint</a></li> + <li><a href="/documentation/programming-guide/#runner-initiated-split">Runner initiated split</a></li> + <li><a href="/documentation/programming-guide/#watermark-estimation">Watermark estimation</a></li> + <li><a href="/documentation/programming-guide/#truncating-during-drain">Truncating during drain</a></li> + <li><a href="/documentation/programming-guide/#bundle-finalization">Bundle finalization</a></li> + </ul> + </li> + + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">Multi-language Pipelines</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/programming-guide/#create-x-lang-transforms">Creating cross-language transforms</a></li> + <li><a href="/documentation/programming-guide/#use-x-lang-transforms">Using cross-language transforms</a></li> + <li><a href="/documentation/programming-guide/#x-lang-transform-runner-support">Runner Support</a></li> + </ul> + </li> + <li><a href="/documentation/programming-guide/#batched-dofns">Batched DoFns</a></li> + </ul> + </li> + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">Pipeline development lifecycle</span> + <ul class="section-nav-list"> + <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li> + <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li> + <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li> + </ul> + </li> + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">Common pipeline patterns</span> + + <ul class="section-nav-list"> + <li><a href="/documentation/patterns/overview/">Overview</a></li> + <li><a href="/documentation/patterns/file-processing/">File processing</a></li> + <li><a href="/documentation/patterns/side-inputs/">Side inputs</a></li> + <li><a href="/documentation/patterns/pipeline-options/">Pipeline options</a></li> + <li><a href="/documentation/patterns/custom-io/">Custom I/O</a></li> + <li><a href="/documentation/patterns/custom-windows/">Custom windows</a></li> + <li><a href="/documentation/patterns/bigqueryio/">BigQueryIO</a></li> + <li><a href="/documentation/patterns/ai-platform/">AI Platform</a></li> + <li><a href="/documentation/patterns/schema/">Schema</a></li> + <li><a href="/documentation/patterns/bqml/">BigQuery ML</a></li> + <li><a href="/documentation/patterns/grouping-elements-for-efficient-external-service-calls/">Grouping elements for efficient external service calls</a></li> + </ul> + </li> + <li class="section-nav-item--collapsible"> + <span class="section-nav-list-title">AI/ML pipelines</span> Review Comment: Note that this section of this page will change in https://github.com/apache/beam/pull/25700/files. If that PR is merged first, add the new topic to whichever category in the left nav corresponds to the table in the Overview page. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
