This is an automated email from the ASF dual-hosted git repository.
Abacn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 8c93ab0efae Update managed-io.md for release 2.74.0-RC1. (#38527)
8c93ab0efae is described below
commit 8c93ab0efaeb04ac8306841d5c0af48cc6af543d
Author: Yi Hu <[email protected]>
AuthorDate: Tue Jun 2 13:33:21 2026 -0400
Update managed-io.md for release 2.74.0-RC1. (#38527)
Co-authored-by: Abacn <actions@GitHub Actions 1018508863.local>
---
.../site/content/en/documentation/io/managed-io.md | 551 +++++++++++----------
1 file changed, 295 insertions(+), 256 deletions(-)
diff --git a/website/www/site/content/en/documentation/io/managed-io.md
b/website/www/site/content/en/documentation/io/managed-io.md
index 46f20c5e11a..ec3bb90183d 100644
--- a/website/www/site/content/en/documentation/io/managed-io.md
+++ b/website/www/site/content/en/documentation/io/managed-io.md
@@ -121,23 +121,25 @@ and Beam SQL is invoked via the Managed API under the
hood.
</td>
<td>
<strong>table</strong> (<code style="color: green">str</code>)<br>
+ autosharding (<code style="color: orange">boolean</code>)<br>
catalog_name (<code style="color: green">str</code>)<br>
catalog_properties (<code>map[<span style="color: green;">str</span>,
<span style="color: green;">str</span>]</code>)<br>
config_properties (<code>map[<span style="color: green;">str</span>,
<span style="color: green;">str</span>]</code>)<br>
direct_write_byte_limit (<code style="color: #f54251">int32</code>)<br>
+ distribution_mode (<code style="color: green">str</code>)<br>
drop (<code>list[<span style="color: green;">str</span>]</code>)<br>
keep (<code>list[<span style="color: green;">str</span>]</code>)<br>
only (<code style="color: green">str</code>)<br>
partition_fields (<code>list[<span style="color:
green;">str</span>]</code>)<br>
+ sort_fields (<code>list[<span style="color:
green;">str</span>]</code>)<br>
table_properties (<code>map[<span style="color: green;">str</span>,
<span style="color: green;">str</span>]</code>)<br>
triggering_frequency_seconds (<code style="color:
#f54251">int32</code>)<br>
</td>
</tr>
<tr>
- <td><strong>MYSQL</strong></td>
+ <td><strong>SQLSERVER</strong></td>
<td>
<strong>jdbc_url</strong> (<code style="color: green">str</code>)<br>
- connection_init_sql (<code>list[<span style="color:
green;">str</span>]</code>)<br>
connection_properties (<code style="color: green">str</code>)<br>
disable_auto_commit (<code style="color: orange">boolean</code>)<br>
fetch_size (<code style="color: #f54251">int32</code>)<br>
@@ -153,7 +155,6 @@ and Beam SQL is invoked via the Managed API under the hood.
<strong>jdbc_url</strong> (<code style="color: green">str</code>)<br>
autosharding (<code style="color: orange">boolean</code>)<br>
batch_size (<code style="color: #f54251">int64</code>)<br>
- connection_init_sql (<code>list[<span style="color:
green;">str</span>]</code>)<br>
connection_properties (<code style="color: green">str</code>)<br>
location (<code style="color: green">str</code>)<br>
password (<code style="color: green">str</code>)<br>
@@ -187,9 +188,28 @@ and Beam SQL is invoked via the Managed API under the hood.
</td>
</tr>
<tr>
- <td><strong>SQLSERVER</strong></td>
+ <td><strong>BIGQUERY</strong></td>
+ <td>
+ kms_key (<code style="color: green">str</code>)<br>
+ query (<code style="color: green">str</code>)<br>
+ row_restriction (<code style="color: green">str</code>)<br>
+ fields (<code>list[<span style="color: green;">str</span>]</code>)<br>
+ table (<code style="color: green">str</code>)<br>
+ </td>
+ <td>
+ <strong>table</strong> (<code style="color: green">str</code>)<br>
+ drop (<code>list[<span style="color: green;">str</span>]</code>)<br>
+ keep (<code>list[<span style="color: green;">str</span>]</code>)<br>
+ kms_key (<code style="color: green">str</code>)<br>
+ only (<code style="color: green">str</code>)<br>
+ triggering_frequency_seconds (<code style="color:
#f54251">int64</code>)<br>
+ </td>
+ </tr>
+ <tr>
+ <td><strong>MYSQL</strong></td>
<td>
<strong>jdbc_url</strong> (<code style="color: green">str</code>)<br>
+ connection_init_sql (<code>list[<span style="color:
green;">str</span>]</code>)<br>
connection_properties (<code style="color: green">str</code>)<br>
disable_auto_commit (<code style="color: orange">boolean</code>)<br>
fetch_size (<code style="color: #f54251">int32</code>)<br>
@@ -205,6 +225,7 @@ and Beam SQL is invoked via the Managed API under the hood.
<strong>jdbc_url</strong> (<code style="color: green">str</code>)<br>
autosharding (<code style="color: orange">boolean</code>)<br>
batch_size (<code style="color: #f54251">int64</code>)<br>
+ connection_init_sql (<code>list[<span style="color:
green;">str</span>]</code>)<br>
connection_properties (<code style="color: green">str</code>)<br>
location (<code style="color: green">str</code>)<br>
password (<code style="color: green">str</code>)<br>
@@ -212,24 +233,6 @@ and Beam SQL is invoked via the Managed API under the hood.
write_statement (<code style="color: green">str</code>)<br>
</td>
</tr>
- <tr>
- <td><strong>BIGQUERY</strong></td>
- <td>
- kms_key (<code style="color: green">str</code>)<br>
- query (<code style="color: green">str</code>)<br>
- row_restriction (<code style="color: green">str</code>)<br>
- fields (<code>list[<span style="color: green;">str</span>]</code>)<br>
- table (<code style="color: green">str</code>)<br>
- </td>
- <td>
- <strong>table</strong> (<code style="color: green">str</code>)<br>
- drop (<code>list[<span style="color: green;">str</span>]</code>)<br>
- keep (<code>list[<span style="color: green;">str</span>]</code>)<br>
- kms_key (<code style="color: green">str</code>)<br>
- only (<code style="color: green">str</code>)<br>
- triggering_frequency_seconds (<code style="color:
#f54251">int64</code>)<br>
- </td>
- </tr>
</table>
</div>
@@ -401,7 +404,7 @@ and Beam SQL is invoked via the Managed API under the hood.
</table>
</div>
-### `KAFKA` Write
+### `KAFKA` Read
<div class="table-container-wrapper">
<table class="table table-bordered">
@@ -418,228 +421,228 @@ and Beam SQL is invoked via the Managed API under the
hood.
<code style="color: green">str</code>
</td>
<td>
- A list of host/port pairs to use for establishing the initial
connection to the Kafka cluster. The client will make use of all servers
irrespective of which servers are specified here for bootstrapping—this list
only impacts the initial hosts used to discover the full set of servers. |
Format: host1:port1,host2:port2,...
+ A list of host/port pairs to use for establishing the initial
connection to the Kafka cluster. The client will make use of all servers
irrespective of which servers are specified here for bootstrapping—this list
only impacts the initial hosts used to discover the full set of servers. This
list should be in the form `host1:port1,host2:port2,...`
</td>
</tr>
<tr>
<td>
- <strong>format</strong>
+ <strong>topic</strong>
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The encoding format for the data stored in Kafka. Valid options are:
RAW,JSON,AVRO,PROTO
+ n/a
</td>
</tr>
<tr>
<td>
- <strong>topic</strong>
+ allow_duplicates
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- n/a
+ If the Kafka read allows duplicates.
</td>
</tr>
<tr>
<td>
- file_descriptor_path
+ confluent_schema_registry_subject
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The path to the Protocol Buffer File Descriptor Set file. This file is
used for schema definition and message serialization.
+ n/a
</td>
</tr>
<tr>
<td>
- message_name
+ confluent_schema_registry_url
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The name of the Protocol Buffer message to be used for schema
extraction and data conversion.
+ n/a
</td>
</tr>
<tr>
<td>
- producer_config_updates
+ consumer_config_updates
</td>
<td>
<code>map[<span style="color: green;">str</span>, <span style="color:
green;">str</span>]</code>
</td>
<td>
- A list of key-value pairs that act as configuration parameters for
Kafka producers. Most of these configurations will not be needed, but if you
need to customize your Kafka producer, you may use this. See a detailed list:
https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html
+ A list of key-value pairs that act as configuration parameters for
Kafka consumers. Most of these configurations will not be needed, but if you
need to customize your Kafka consumer, you may use this. See a detailed list:
https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html
</td>
</tr>
<tr>
<td>
- schema
+ file_descriptor_path
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- n/a
+ The path to the Protocol Buffer File Descriptor Set file. This file is
used for schema definition and message serialization.
</td>
</tr>
- </table>
-</div>
-
-### `KAFKA` Read
-
-<div class="table-container-wrapper">
- <table class="table table-bordered">
- <tr>
- <th>Configuration</th>
- <th>Type</th>
- <th>Description</th>
- </tr>
<tr>
<td>
- <strong>bootstrap_servers</strong>
+ format
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- A list of host/port pairs to use for establishing the initial
connection to the Kafka cluster. The client will make use of all servers
irrespective of which servers are specified here for bootstrapping—this list
only impacts the initial hosts used to discover the full set of servers. This
list should be in the form `host1:port1,host2:port2,...`
+ The encoding format for the data stored in Kafka. Valid options are:
RAW,STRING,AVRO,JSON,PROTO
</td>
</tr>
<tr>
<td>
- <strong>topic</strong>
+ message_name
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- n/a
+ The name of the Protocol Buffer message to be used for schema
extraction and data conversion.
</td>
</tr>
<tr>
<td>
- allow_duplicates
+ offset_deduplication
</td>
<td>
<code style="color: orange">boolean</code>
</td>
<td>
- If the Kafka read allows duplicates.
+ If the redistribute is using offset deduplication mode.
</td>
</tr>
<tr>
<td>
- confluent_schema_registry_subject
+ redistribute_by_record_key
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- n/a
+ If the redistribute keys by the Kafka record key.
</td>
</tr>
<tr>
<td>
- confluent_schema_registry_url
+ redistribute_num_keys
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int32</code>
</td>
<td>
- n/a
+ The number of keys for redistributing Kafka inputs.
</td>
</tr>
<tr>
<td>
- consumer_config_updates
+ redistributed
</td>
<td>
- <code>map[<span style="color: green;">str</span>, <span style="color:
green;">str</span>]</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- A list of key-value pairs that act as configuration parameters for
Kafka consumers. Most of these configurations will not be needed, but if you
need to customize your Kafka consumer, you may use this. See a detailed list:
https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html
+ If the Kafka read should be redistributed.
</td>
</tr>
<tr>
<td>
- file_descriptor_path
+ schema
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The path to the Protocol Buffer File Descriptor Set file. This file is
used for schema definition and message serialization.
+ The schema in which the data is encoded in the Kafka topic. For AVRO
data, this is a schema defined with AVRO schema syntax
(https://avro.apache.org/docs/1.10.2/spec.html#schemas). For JSON data, this is
a schema defined with JSON-schema syntax (https://json-schema.org/). If a URL
to Confluent Schema Registry is provided, then this field is ignored, and the
schema is fetched from Confluent Schema Registry.
</td>
</tr>
+ </table>
+</div>
+
+### `KAFKA` Write
+
+<div class="table-container-wrapper">
+ <table class="table table-bordered">
+ <tr>
+ <th>Configuration</th>
+ <th>Type</th>
+ <th>Description</th>
+ </tr>
<tr>
<td>
- format
+ <strong>bootstrap_servers</strong>
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The encoding format for the data stored in Kafka. Valid options are:
RAW,STRING,AVRO,JSON,PROTO
+ A list of host/port pairs to use for establishing the initial
connection to the Kafka cluster. The client will make use of all servers
irrespective of which servers are specified here for bootstrapping—this list
only impacts the initial hosts used to discover the full set of servers. |
Format: host1:port1,host2:port2,...
</td>
</tr>
<tr>
<td>
- message_name
+ <strong>format</strong>
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The name of the Protocol Buffer message to be used for schema
extraction and data conversion.
+ The encoding format for the data stored in Kafka. Valid options are:
RAW,JSON,AVRO,PROTO
</td>
</tr>
<tr>
<td>
- offset_deduplication
+ <strong>topic</strong>
</td>
<td>
- <code style="color: orange">boolean</code>
+ <code style="color: green">str</code>
</td>
<td>
- If the redistribute is using offset deduplication mode.
+ n/a
</td>
</tr>
<tr>
<td>
- redistribute_by_record_key
+ file_descriptor_path
</td>
<td>
- <code style="color: orange">boolean</code>
+ <code style="color: green">str</code>
</td>
<td>
- If the redistribute keys by the Kafka record key.
+ The path to the Protocol Buffer File Descriptor Set file. This file is
used for schema definition and message serialization.
</td>
</tr>
<tr>
<td>
- redistribute_num_keys
+ message_name
</td>
<td>
- <code style="color: #f54251">int32</code>
+ <code style="color: green">str</code>
</td>
<td>
- The number of keys for redistributing Kafka inputs.
+ The name of the Protocol Buffer message to be used for schema
extraction and data conversion.
</td>
</tr>
<tr>
<td>
- redistributed
+ producer_config_updates
</td>
<td>
- <code style="color: orange">boolean</code>
+ <code>map[<span style="color: green;">str</span>, <span style="color:
green;">str</span>]</code>
</td>
<td>
- If the Kafka read should be redistributed.
+ A list of key-value pairs that act as configuration parameters for
Kafka producers. Most of these configurations will not be needed, but if you
need to customize your Kafka producer, you may use this. See a detailed list:
https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html
</td>
</tr>
<tr>
@@ -650,7 +653,7 @@ and Beam SQL is invoked via the Managed API under the hood.
<code style="color: green">str</code>
</td>
<td>
- The schema in which the data is encoded in the Kafka topic. For AVRO
data, this is a schema defined with AVRO schema syntax
(https://avro.apache.org/docs/1.10.2/spec.html#schemas). For JSON data, this is
a schema defined with JSON-schema syntax (https://json-schema.org/). If a URL
to Confluent Schema Registry is provided, then this field is ignored, and the
schema is fetched from Confluent Schema Registry.
+ n/a
</td>
</tr>
</table>
@@ -765,6 +768,17 @@ and Beam SQL is invoked via the Managed API under the hood.
A fully-qualified table identifier. You may also provide a template to
write to multiple dynamic destinations, for example:
`dataset.my_{col1}_{col2.nested}_table`.
</td>
</tr>
+ <tr>
+ <td>
+ autosharding
+ </td>
+ <td>
+ <code style="color: orange">boolean</code>
+ </td>
+ <td>
+ Enables dynamic sharding to automatically adjust the number of
parallel writers based on data volume. It handles data skew by further
sub-dividing partitions into multiple shards to prevent bottlenecks during
high-throughput writes. Only available with 'hash' distribution mode.
+ </td>
+ </tr>
<tr>
<td>
catalog_name
@@ -809,6 +823,19 @@ and Beam SQL is invoked via the Managed API under the hood.
For a streaming pipeline, sets the limit for lifting bundles into the
direct write path.
</td>
</tr>
+ <tr>
+ <td>
+ distribution_mode
+ </td>
+ <td>
+ <code style="color: green">str</code>
+ </td>
+ <td>
+ Defines distribution of write data. Supported distributions:
+- none: don't shuffle rows (default)
+- hash: shuffle rows by partition key before writing data
+ </td>
+ </tr>
<tr>
<td>
drop
@@ -864,6 +891,18 @@ and Beam SQL is invoked via the Managed API under the hood.
For more information on partition transforms, please visit
https://iceberg.apache.org/spec/#partition-transforms.
</td>
</tr>
+ <tr>
+ <td>
+ sort_fields
+ </td>
+ <td>
+ <code>list[<span style="color: green;">str</span>]</code>
+ </td>
+ <td>
+ Fields used to set the table's sort order, applied when the table is
created. Each entry has the form `<term> [asc|desc] [nulls first|nulls last]`,
where `<term>` is a field name or one of the partition transforms (e.g.
`bucket(col, 4)`, `day(ts)`). Direction defaults to ascending; null order
defaults to nulls-first for ascending and nulls-last for descending. Note: this
sets the table's declared sort order as metadata; it does not cause Beam to
physically sort records before writing.
+For more information on sort orders, please visit
https://iceberg.apache.org/spec/#sort-orders.
+ </td>
+ </tr>
<tr>
<td>
table_properties
@@ -890,7 +929,7 @@ For more information on table properties, please visit
https://iceberg.apache.or
</table>
</div>
-### `MYSQL` Read
+### `SQLSERVER` Read
<div class="table-container-wrapper">
<table class="table table-bordered">
@@ -910,17 +949,6 @@ For more information on table properties, please visit
https://iceberg.apache.or
Connection URL for the JDBC source.
</td>
</tr>
- <tr>
- <td>
- connection_init_sql
- </td>
- <td>
- <code>list[<span style="color: green;">str</span>]</code>
- </td>
- <td>
- Sets the connection init sql statements used by the Driver. Only MySQL
and MariaDB support this.
- </td>
- </tr>
<tr>
<td>
connection_properties
@@ -1034,7 +1062,7 @@ For more information on table properties, please visit
https://iceberg.apache.or
</table>
</div>
-### `MYSQL` Write
+### `SQLSERVER` Write
<div class="table-container-wrapper">
<table class="table table-bordered">
@@ -1076,17 +1104,6 @@ For more information on table properties, please visit
https://iceberg.apache.or
n/a
</td>
</tr>
- <tr>
- <td>
- connection_init_sql
- </td>
- <td>
- <code>list[<span style="color: green;">str</span>]</code>
- </td>
- <td>
- Sets the connection init sql statements used by the Driver. Only MySQL
and MariaDB support this.
- </td>
- </tr>
<tr>
<td>
connection_properties
@@ -1145,7 +1162,7 @@ For more information on table properties, please visit
https://iceberg.apache.or
</table>
</div>
-### `POSTGRES` Write
+### `POSTGRES` Read
<div class="table-container-wrapper">
<table class="table table-bordered">
@@ -1162,173 +1179,173 @@ For more information on table properties, please
visit https://iceberg.apache.or
<code style="color: green">str</code>
</td>
<td>
- Connection URL for the JDBC sink.
+ Connection URL for the JDBC source.
</td>
</tr>
<tr>
<td>
- autosharding
+ connection_properties
</td>
<td>
- <code style="color: orange">boolean</code>
+ <code style="color: green">str</code>
</td>
<td>
- If true, enables using a dynamically determined number of shards to
write.
+ Used to set connection properties passed to the JDBC driver not
already defined as standalone parameter (e.g. username and password can be set
using parameters above accordingly). Format of the string must be
"key1=value1;key2=value2;".
</td>
</tr>
<tr>
<td>
- batch_size
+ fetch_size
</td>
<td>
- <code style="color: #f54251">int64</code>
+ <code style="color: #f54251">int32</code>
</td>
<td>
- n/a
+ This method is used to override the size of the data that is going to
be fetched and loaded in memory per every database call. It should ONLY be used
if the default value throws memory errors.
</td>
</tr>
<tr>
<td>
- connection_properties
+ location
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Used to set connection properties passed to the JDBC driver not
already defined as standalone parameter (e.g. username and password can be set
using parameters above accordingly). Format of the string must be
"key1=value1;key2=value2;".
+ Name of the table to read from.
</td>
</tr>
<tr>
<td>
- location
+ num_partitions
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int32</code>
</td>
<td>
- Name of the table to write to.
+ The number of partitions
</td>
</tr>
<tr>
<td>
- password
+ output_parallelization
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- Password for the JDBC source.
+ Whether to reshuffle the resulting PCollection so results are
distributed to all workers.
</td>
</tr>
<tr>
<td>
- username
+ partition_column
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Username for the JDBC source.
+ Name of a column of numeric type that will be used for partitioning.
</td>
</tr>
<tr>
<td>
- write_statement
+ password
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- SQL query used to insert records into the JDBC sink.
+ Password for the JDBC source.
</td>
</tr>
- </table>
-</div>
-
-### `POSTGRES` Read
-
-<div class="table-container-wrapper">
- <table class="table table-bordered">
- <tr>
- <th>Configuration</th>
- <th>Type</th>
- <th>Description</th>
- </tr>
<tr>
<td>
- <strong>jdbc_url</strong>
+ read_query
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Connection URL for the JDBC source.
+ SQL query used to query the JDBC source.
</td>
</tr>
<tr>
<td>
- connection_properties
+ username
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Used to set connection properties passed to the JDBC driver not
already defined as standalone parameter (e.g. username and password can be set
using parameters above accordingly). Format of the string must be
"key1=value1;key2=value2;".
+ Username for the JDBC source.
</td>
</tr>
+ </table>
+</div>
+
+### `POSTGRES` Write
+
+<div class="table-container-wrapper">
+ <table class="table table-bordered">
+ <tr>
+ <th>Configuration</th>
+ <th>Type</th>
+ <th>Description</th>
+ </tr>
<tr>
<td>
- fetch_size
+ <strong>jdbc_url</strong>
</td>
<td>
- <code style="color: #f54251">int32</code>
+ <code style="color: green">str</code>
</td>
<td>
- This method is used to override the size of the data that is going to
be fetched and loaded in memory per every database call. It should ONLY be used
if the default value throws memory errors.
+ Connection URL for the JDBC sink.
</td>
</tr>
<tr>
<td>
- location
+ autosharding
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- Name of the table to read from.
+ If true, enables using a dynamically determined number of shards to
write.
</td>
</tr>
<tr>
<td>
- num_partitions
+ batch_size
</td>
<td>
- <code style="color: #f54251">int32</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- The number of partitions
+ n/a
</td>
</tr>
<tr>
<td>
- output_parallelization
+ connection_properties
</td>
<td>
- <code style="color: orange">boolean</code>
+ <code style="color: green">str</code>
</td>
<td>
- Whether to reshuffle the resulting PCollection so results are
distributed to all workers.
+ Used to set connection properties passed to the JDBC driver not
already defined as standalone parameter (e.g. username and password can be set
using parameters above accordingly). Format of the string must be
"key1=value1;key2=value2;".
</td>
</tr>
<tr>
<td>
- partition_column
+ location
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Name of a column of numeric type that will be used for partitioning.
+ Name of the table to write to.
</td>
</tr>
<tr>
@@ -1344,30 +1361,30 @@ For more information on table properties, please visit
https://iceberg.apache.or
</tr>
<tr>
<td>
- read_query
+ username
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- SQL query used to query the JDBC source.
+ Username for the JDBC source.
</td>
</tr>
<tr>
<td>
- username
+ write_statement
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Username for the JDBC source.
+ SQL query used to insert records into the JDBC sink.
</td>
</tr>
</table>
</div>
-### `SQLSERVER` Read
+### `BIGQUERY` Write
<div class="table-container-wrapper">
<table class="table table-bordered">
@@ -1378,129 +1395,141 @@ For more information on table properties, please
visit https://iceberg.apache.or
</tr>
<tr>
<td>
- <strong>jdbc_url</strong>
+ <strong>table</strong>
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Connection URL for the JDBC source.
+ The bigquery table to write to. Format:
[${PROJECT}:]${DATASET}.${TABLE}
</td>
</tr>
<tr>
<td>
- connection_properties
+ drop
</td>
<td>
- <code style="color: green">str</code>
+ <code>list[<span style="color: green;">str</span>]</code>
</td>
<td>
- Used to set connection properties passed to the JDBC driver not
already defined as standalone parameter (e.g. username and password can be set
using parameters above accordingly). Format of the string must be
"key1=value1;key2=value2;".
+ A list of field names to drop from the input record before writing. Is
mutually exclusive with 'keep' and 'only'.
</td>
</tr>
<tr>
<td>
- disable_auto_commit
+ keep
</td>
<td>
- <code style="color: orange">boolean</code>
+ <code>list[<span style="color: green;">str</span>]</code>
</td>
<td>
- Whether to disable auto commit on read. Defaults to true if not
provided. The need for this config varies depending on the database platform.
Informix requires this to be set to false while Postgres requires this to be
set to true.
+ A list of field names to keep in the input record. All other fields
are dropped before writing. Is mutually exclusive with 'drop' and 'only'.
</td>
</tr>
<tr>
<td>
- fetch_size
+ kms_key
</td>
<td>
- <code style="color: #f54251">int32</code>
+ <code style="color: green">str</code>
</td>
<td>
- This method is used to override the size of the data that is going to
be fetched and loaded in memory per every database call. It should ONLY be used
if the default value throws memory errors.
+ Use this Cloud KMS key to encrypt your data
</td>
</tr>
<tr>
<td>
- location
+ only
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Name of the table to read from.
+ The name of a single record field that should be written. Is mutually
exclusive with 'keep' and 'drop'.
</td>
</tr>
<tr>
<td>
- num_partitions
+ triggering_frequency_seconds
</td>
<td>
- <code style="color: #f54251">int32</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- The number of partitions
+ Determines how often to 'commit' progress into BigQuery. Default is
every 5 seconds.
</td>
</tr>
+ </table>
+</div>
+
+### `BIGQUERY` Read
+
+<div class="table-container-wrapper">
+ <table class="table table-bordered">
+ <tr>
+ <th>Configuration</th>
+ <th>Type</th>
+ <th>Description</th>
+ </tr>
<tr>
<td>
- output_parallelization
+ kms_key
</td>
<td>
- <code style="color: orange">boolean</code>
+ <code style="color: green">str</code>
</td>
<td>
- Whether to reshuffle the resulting PCollection so results are
distributed to all workers.
+ Use this Cloud KMS key to encrypt your data
</td>
</tr>
<tr>
<td>
- partition_column
+ query
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Name of a column of numeric type that will be used for partitioning.
+ The SQL query to be executed to read from the BigQuery table.
</td>
</tr>
<tr>
<td>
- password
+ row_restriction
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Password for the JDBC source.
+ Read only rows that match this filter, which must be compatible with
Google standard SQL. This is not supported when reading via query.
</td>
</tr>
<tr>
<td>
- read_query
+ fields
</td>
<td>
- <code style="color: green">str</code>
+ <code>list[<span style="color: green;">str</span>]</code>
</td>
<td>
- SQL query used to query the JDBC source.
+ Read only the specified fields (columns) from a BigQuery table. Fields
may not be returned in the order specified. If no value is specified, then all
fields are returned. Example: "col1, col2, col3"
</td>
</tr>
<tr>
<td>
- username
+ table
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Username for the JDBC source.
+ The fully-qualified name of the BigQuery table to read from. Format:
[${PROJECT}:]${DATASET}.${TABLE}
</td>
</tr>
</table>
</div>
-### `SQLSERVER` Write
+### `MYSQL` Read
<div class="table-container-wrapper">
<table class="table table-bordered">
@@ -1517,29 +1546,18 @@ For more information on table properties, please visit
https://iceberg.apache.or
<code style="color: green">str</code>
</td>
<td>
- Connection URL for the JDBC sink.
- </td>
- </tr>
- <tr>
- <td>
- autosharding
- </td>
- <td>
- <code style="color: orange">boolean</code>
- </td>
- <td>
- If true, enables using a dynamically determined number of shards to
write.
+ Connection URL for the JDBC source.
</td>
</tr>
<tr>
<td>
- batch_size
+ connection_init_sql
</td>
<td>
- <code style="color: #f54251">int64</code>
+ <code>list[<span style="color: green;">str</span>]</code>
</td>
<td>
- n/a
+ Sets the connection init sql statements used by the Driver. Only MySQL
and MariaDB support this.
</td>
</tr>
<tr>
@@ -1555,119 +1573,107 @@ For more information on table properties, please
visit https://iceberg.apache.or
</tr>
<tr>
<td>
- location
+ disable_auto_commit
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- Name of the table to write to.
+ Whether to disable auto commit on read. Defaults to true if not
provided. The need for this config varies depending on the database platform.
Informix requires this to be set to false while Postgres requires this to be
set to true.
</td>
</tr>
<tr>
<td>
- password
+ fetch_size
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int32</code>
</td>
<td>
- Password for the JDBC source.
+ This method is used to override the size of the data that is going to
be fetched and loaded in memory per every database call. It should ONLY be used
if the default value throws memory errors.
</td>
</tr>
<tr>
<td>
- username
+ location
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Username for the JDBC source.
+ Name of the table to read from.
</td>
</tr>
<tr>
<td>
- write_statement
+ num_partitions
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int32</code>
</td>
<td>
- SQL query used to insert records into the JDBC sink.
+ The number of partitions
</td>
</tr>
- </table>
-</div>
-
-### `BIGQUERY` Read
-
-<div class="table-container-wrapper">
- <table class="table table-bordered">
- <tr>
- <th>Configuration</th>
- <th>Type</th>
- <th>Description</th>
- </tr>
<tr>
<td>
- kms_key
+ output_parallelization
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- Use this Cloud KMS key to encrypt your data
+ Whether to reshuffle the resulting PCollection so results are
distributed to all workers.
</td>
</tr>
<tr>
<td>
- query
+ partition_column
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The SQL query to be executed to read from the BigQuery table.
+ Name of a column of numeric type that will be used for partitioning.
</td>
</tr>
<tr>
<td>
- row_restriction
+ password
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Read only rows that match this filter, which must be compatible with
Google standard SQL. This is not supported when reading via query.
+ Password for the JDBC source.
</td>
</tr>
<tr>
<td>
- fields
+ read_query
</td>
<td>
- <code>list[<span style="color: green;">str</span>]</code>
+ <code style="color: green">str</code>
</td>
<td>
- Read only the specified fields (columns) from a BigQuery table. Fields
may not be returned in the order specified. If no value is specified, then all
fields are returned. Example: "col1, col2, col3"
+ SQL query used to query the JDBC source.
</td>
</tr>
<tr>
<td>
- table
+ username
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The fully-qualified name of the BigQuery table to read from. Format:
[${PROJECT}:]${DATASET}.${TABLE}
+ Username for the JDBC source.
</td>
</tr>
</table>
</div>
-### `BIGQUERY` Write
+### `MYSQL` Write
<div class="table-container-wrapper">
<table class="table table-bordered">
@@ -1678,68 +1684,101 @@ For more information on table properties, please visit
https://iceberg.apache.or
</tr>
<tr>
<td>
- <strong>table</strong>
+ <strong>jdbc_url</strong>
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The bigquery table to write to. Format:
[${PROJECT}:]${DATASET}.${TABLE}
+ Connection URL for the JDBC sink.
</td>
</tr>
<tr>
<td>
- drop
+ autosharding
</td>
<td>
- <code>list[<span style="color: green;">str</span>]</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- A list of field names to drop from the input record before writing. Is
mutually exclusive with 'keep' and 'only'.
+ If true, enables using a dynamically determined number of shards to
write.
</td>
</tr>
<tr>
<td>
- keep
+ batch_size
+ </td>
+ <td>
+ <code style="color: #f54251">int64</code>
+ </td>
+ <td>
+ n/a
+ </td>
+ </tr>
+ <tr>
+ <td>
+ connection_init_sql
</td>
<td>
<code>list[<span style="color: green;">str</span>]</code>
</td>
<td>
- A list of field names to keep in the input record. All other fields
are dropped before writing. Is mutually exclusive with 'drop' and 'only'.
+ Sets the connection init sql statements used by the Driver. Only MySQL
and MariaDB support this.
</td>
</tr>
<tr>
<td>
- kms_key
+ connection_properties
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Use this Cloud KMS key to encrypt your data
+ Used to set connection properties passed to the JDBC driver not
already defined as standalone parameter (e.g. username and password can be set
using parameters above accordingly). Format of the string must be
"key1=value1;key2=value2;".
</td>
</tr>
<tr>
<td>
- only
+ location
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- The name of a single record field that should be written. Is mutually
exclusive with 'keep' and 'drop'.
+ Name of the table to write to.
</td>
</tr>
<tr>
<td>
- triggering_frequency_seconds
+ password
</td>
<td>
- <code style="color: #f54251">int64</code>
+ <code style="color: green">str</code>
</td>
<td>
- Determines how often to 'commit' progress into BigQuery. Default is
every 5 seconds.
+ Password for the JDBC source.
+ </td>
+ </tr>
+ <tr>
+ <td>
+ username
+ </td>
+ <td>
+ <code style="color: green">str</code>
+ </td>
+ <td>
+ Username for the JDBC source.
+ </td>
+ </tr>
+ <tr>
+ <td>
+ write_statement
+ </td>
+ <td>
+ <code style="color: green">str</code>
+ </td>
+ <td>
+ SQL query used to insert records into the JDBC sink.
</td>
</tr>
</table>
