bullet03 commented on a change in pull request #17040: URL: https://github.com/apache/beam/pull/17040#discussion_r826865751
########## File path: website/www/site/content/en/documentation/io/connectors.md ########## @@ -0,0 +1,940 @@ +--- +title: "I/O Connectors" +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# I/O Connectors + +Apache Beam I/O connectors provide read and write transforms for the most popular data storage systems so that Beam users can benefit from native optimised connectivity. With the available I/Os, Apache Beam pipelines can read and write data from and to an external storage type in a unified and distributed way. + +## Built-in I/O Connectors + +This table provides a consolidated, at-a-glance overview of the available built-in I/O connectors. +<div class="table-container-wrapper"> + <table class="table table-bordered"> + <tr> + <th>Connector Name</th> + <th>Source Supported</th> + <th>Sink Supported</th> + <th>Java</th> + <th>Python</th> + <th>Go</th> + <th>Batch Supported</th> + <th>Streaming Supported</th> + </tr> + <tr> + <td>FileIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/FileIO.html";>native</a> + </td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/pydoc/2.36.0/apache_beam.io.fileio.html";>native</a> + </td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="present">✔</td> + </tr> + <tr> + <td>AvroIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/AvroIO.html";>native</a> + </td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/pydoc/2.36.0/apache_beam.io.avroio.html";>native</a> + </td> + <td class="present"> + ✔ + <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/avroio";>native</a> + </td> + <td class="present">✔</td> + <td class="present">✔</td> + </tr> + <tr> + <td>TextIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/TextIO.html";>native</a> + </td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/pydoc/2.36.0/apache_beam.io.textio.html";>native</a> + </td> + <td class="present"> + ✔ + <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/textio";>native</a> + </td> + <td class="present">✔</td> + <td class="present">✔</td> + </tr> + <tr> + <td>TFRecordIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/TFRecordIO.html";>native</a> + </td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/pydoc/2.36.0/apache_beam.io.tfrecordio.html";>native</a> + </td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="absent">✘</td> + </tr> + <tr> + <td>XmlIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/xml/XmlIO.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="absent">✘</td> + </tr> + <tr> + <td>TikaIO</td> + <td class="present">✔</td> + <td class="absent">✘</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/tika/TikaIO.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="present">✔</td> + </tr> + <tr> + <td>ParquetIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/parquet/ParquetIO.html";>native</a> + </td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/parquet/ParquetIO.html";>native</a> + </td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="absent">✘</td> + </tr> + <tr> + <td>ThriftIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/thrift/ThriftIO.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="absent">✘</td> + </tr> + <tr> + <td>S3IO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td>via X-language</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/pydoc/2.36.0/apache_beam.io.aws.s3io.html";>native</a> + </td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="absent">✘</td> + </tr> + <tr> + <td>GcsIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td>via X-language</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/pydoc/2.36.0/apache_beam.io.gcp.gcsio.html";>native</a> + </td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="absent">✘</td> + </tr> + <tr> + <td>HadoopFileSystem</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.html";>native</a> + </td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/pydoc/2.36.0/apache_beam.io.hadoopfilesystem.html";>native</a> + </td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="absent">✘</td> + </tr> + <tr> + <td>GcsFileSystem</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.html";>native</a> + </td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/pydoc/2.36.0/apache_beam.io.gcp.gcsfilesystem.html";>native</a> + </td> + <td class="present"> + ✔ + <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/filesystem/gcs";>native</a> + </td> + <td class="present">✔</td> + <td class="absent">✘</td> + </tr> + <tr> + <td>LocalFileSystem</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/LocalFileSystemRegistrar.html";>native</a> + </td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/LocalFileSystemRegistrar.html";>native</a> + </td> + <td class="present"> + ✔ + <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/filesystem/local";>native</a> + </td> + <td class="present">✔</td> + <td class="absent">✘</td> + </tr> + <tr> + <td>S3FileSystem</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/aws/s3/S3FileSystemRegistrar.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="absent">✘</td> + </tr> + <tr> + <td>In-memory</td> + <td class="absent">✘</td> + <td class="present">✔</td> + <td class="absent">✘</td> + <td class="absent">✘</td> + <td class="present"> + ✔ + <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/filesystem/memfs";>native</a> + </td> + <td class="present">✔</td> + <td class="absent">✘</td> + </tr> + <tr> + <td>KinesisIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/kinesis/KinesisIO.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="present">✔</td> + </tr> + <tr> + <td>AmqpIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/amqp/AmqpIO.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="present">✔</td> + </tr> + <tr> + <td>KafkaIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/kafka/KafkaIO.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="present">✔</td> + </tr> + <tr> + <td>PubSubIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html";>native</a> + </td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/pydoc/2.36.0/apache_beam.io.gcp.pubsub.html";>native</a> + </td> + <td class="present"> + ✔ + <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/pubsubio";>native</a> + </td> + <td class="present">✔</td> + <td class="present">✔</td> + </tr> + <tr> + <td>JmsIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/jms/JmsIO.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="present">✔</td> + </tr> + <tr> + <td>MqttIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/mqtt/MqttIO.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="present">✔</td> + </tr> + <tr> + <td>RabbitMqIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="present">✔</td> + </tr> + <tr> + <td>SqsIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/aws2/sqs/SqsIO.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="present">✔</td> + </tr> + <tr> + <td>SnsIO</td> + <td class="absent">✘</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/aws2/sns/SnsIO.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="absent">✘</td> + </tr> + <tr> + <td>CassandraIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/cassandra/CassandraIO.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="absent">✘</td> + </tr> + <tr> + <td>HadoopFormatIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="present">✔</td> + </tr> + <tr> + <td>HBaseIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/hbase/HBaseIO.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="absent">✘</td> + </tr> + <tr> + <td>HCatalogIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href="https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/hcatalog/HCatalogIO.html";>native</a> + </td> + <td>via X-language</td> + <td>via X-language</td> + <td class="present">✔</td> + <td class="present">✔</td> + </tr> + <tr> + <td>KuduIO</td> + <td class="present">✔</td> + <td class="present">✔</td> + <td class="present"> + ✔ + <a href=https://beam.apache.org/releases/javadoc/2.36.0/org/apache/beam/sdk/io/kudu/KuduIO.html";>native</a> Review comment: done, tested all the links -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
