[ https://issues.apache.org/jira/browse/BEAM-9916?focusedWorklogId=437309&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437309 ]
ASF GitHub Bot logged work on BEAM-9916: ---------------------------------------- Author: ASF GitHub Bot Created on: 27/May/20 03:32 Start Date: 27/May/20 03:32 Worklog Time Spent: 10m Work Description: pabloem commented on a change in pull request #11802: URL: https://github.com/apache/beam/pull/11802#discussion_r430712882 ########## File path: website/www/site/data/io_matrix.yaml ########## @@ -0,0 +1,377 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +categories: + - name: File-based + description: These I/O connectors involve working with files. + rows: + - transform: FileIO + description: "General-purpose transforms for working with files: listing files (matching), reading and writing." + implementations: + - language: java + name: org.apache.beam.sdk.io.FileIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.html + - language: py + name: apache_beam.io.FileIO + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.fileio.html + - transform: AvroIO + description: PTransforms for reading from and writing to [Avro](https://avro.apache.org/) files. + implementations: + - language: java + name: org.apache.beam.sdk.io.AvroIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/AvroIO.html + - language: py + name: apache_beam.io.avroio + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.avroio.html + - language: go + name: github.com/apache/beam/sdks/go/pkg/beam/io/avroio + url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/avroio + - transform: TextIO + description: PTransforms for reading and writing text files. + implementations: + - language: java + name: org.apache.beam.sdk.io.TextIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html + - language: py + name: apache_beam.io.textio + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html + - language: go + name: github.com/apache/beam/sdks/go/pkg/beam/io/textio + url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/textio + - transform: TFRecordIO + description: PTransforms for reading and writing [TensorFlow TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) files. + implementations: + - language: java + name: org.apache.beam.sdk.io.TFRecordIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TFRecordIO.html + - language: py + name: apache_beam.io.tfrecordio + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.tfrecordio.html + - transform: XmlIO + description: Transforms for reading and writing XML files using [JAXB](https://www.oracle.com/technical-resources/articles/javase/jaxb.html) mappers. + implementations: + - language: java + name: org.apache.beam.sdk.io.xml.XmlIO + url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/xml/XmlIO.html + - transform: TikaIO + description: Transforms for parsing arbitrary files using [Apache Tika](https://tika.apache.org/). + implementations: + - language: java + name: org.apache.beam.sdk.io.tika.TikaIO + url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/tika/TikaIO.html + - transform: ParquetIO + description: IO for reading from and writing to [Parquet](https://parquet.apache.org/) files. + docs: /documentation/io/built-in/parquet/ + implementations: + - language: java + name: org.apache.beam.sdk.io.parquet.ParquetIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/parquet/ParquetIO.html + - language: py + name: apache_beam.io.parquetio + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.parquetio.html + - transform: ThriftIO + description: PTransforms for reading and writing files containing [Thrift](https://thrift.apache.org/)-encoded data. + implementations: + - language: java + name: org.apache.beam.sdk.io.thrift.ThriftIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/thrift/ThriftIO.html + - transform: VcfIO + description: A source for reading from [VCF files](https://samtools.github.io/hts-specs/VCFv4.2.pdf) (version 4.x). + implementations: + - language: py + name: apache_beam.io.vcfio + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.vcfio.html + - transform: S3IO + description: A source for reading from and writing to [Amazon S3](https://aws.amazon.com/s3/). + implementations: + - language: py + name: apache_beam.io.aws.s3io + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.aws.s3io.html + - transform: GcsIO + description: A source for reading from and writing to [Google Cloud Storage](https://cloud.google.com/storage). + implementations: + - language: py + name: apache_beam.io.gcp.gcsio + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsio.html + - name: FileSystem + description: Beam provides a File system interface that defines APIs for writing file systems agnostic code. Several I/O connectors are implemented as a FileSystem implementation. + rows: + - transform: HadoopFileSystem + description: "`FileSystem` implementation for accessing [Hadoop](https://hadoop.apache.org/) Distributed File System files." + implementations: + - language: java + name: org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrar + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.html + - language: py + name: apache_beam.io.hadoopfilesystem + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.hadoopfilesystem.html + - transform: GcsFileSystem + description: "`FileSystem` implementation for [Google Cloud Storage](https://cloud.google.com/storage)." + implementations: + - language: java + name: org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.html + - language: py + name: apache_beam.io.gcp.gcsfilesystem + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsfilesystem.html + - language: go + name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs + url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs + - transform: LocalFileSystem + description: "`FileSystem` implementation for accessing files on disk." + implementations: + - language: java + name: org.apache.beam.sdk.io.LocalFileSystemRegistrar + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/LocalFileSystemRegistrar.html + - language: py + name: apache_beam.io.localfilesystem + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.localfilesystem.html + - language: go + name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local + url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local + - transform: S3FileSystem + description: "`FileSystem` implementation for [Amazon S3](https://aws.amazon.com/s3/)." + implementations: + - language: java + name: org.apache.beam.sdk.io.aws.s3.S3FileSystemRegistrar + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/package-summary.html + - transform: In-memory + description: "`FileSystem` implementation in memory; useful for testing." + implementations: + - language: go + name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs + url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs + - name: Messaging + description: These I/O connectors typically involve working with unbounded sources that come from messaging sources. + rows: + - transform: KinesisIO + description: PTransforms for reading from and writing to [Kinesis](https://aws.amazon.com/kinesis/) streams. + implementations: + - language: java + name: org.apache.beam.sdk.io.kinesis.KinesisIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kinesis/KinesisIO.html + - transform: AmqpIO + description: AMQP 1.0 protocol using the Apache QPid Proton-J library + implementations: + - language: java + name: org.apache.beam.sdk.io.amqp.AmqpIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/amqp/AmqpIO.html + - transform: KafkaIO + description: Read and Write PTransforms for [Apache Kafka](https://kafka.apache.org/). + implementations: + - language: java + name: org.apache.beam.sdk.io.kafka.KafkaIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kafka/KafkaIO.html + - language: py + name: apache_beam.io.external.kafka + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.kafka.html + - transform: PubSubIO + description: Read and Write PTransforms for [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) streams. + implementations: + - language: java + name: org.apache.beam.sdk.io.gcp.pubsub.PubsubIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html + - language: py + name: apache_beam.io.gcp.pubsub + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.pubsub.html + - language: py + name: apache_beam.io.external.gcp.pubsub + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.gcp.pubsub.html + - language: go + name: github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio + url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio + - transform: JmsIO + description: An unbounded source for [JMS](https://www.oracle.com/java/technologies/java-message-service.html) destinations (queues or topics). + implementations: + - language: java + name: org.apache.beam.sdk.io.jms.JmsIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jms/JmsIO.html + - transform: MqttIO + description: An unbounded source for [MQTT](https://mqtt.org/) broker. + implementations: + - language: java + name: org.apache.beam.sdk.io.mqtt.MqttIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/mqtt/MqttIO.html + - transform: RabbitMqIO + description: A IO to publish or consume messages with a RabbitMQ broker. + implementations: + - language: java + name: org.apache.beam.sdk.io.rabbitmq.RabbitMqIO + url: https://github.com/apache/beam/blob/master/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java Review comment: it seems that javadoc export is deactivated for those IOs (see line 20 of `sdks/java/io/kudu/build.gradle` and `sdks/java/io/rabbitmq/build.gradle`) - @aromanenko-dev do you know if that's expected or should be fixed? ########## File path: website/www/site/data/io_matrix.yaml ########## @@ -0,0 +1,377 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +categories: + - name: File-based + description: These I/O connectors involve working with files. + rows: + - transform: FileIO + description: "General-purpose transforms for working with files: listing files (matching), reading and writing." + implementations: + - language: java + name: org.apache.beam.sdk.io.FileIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.html + - language: py + name: apache_beam.io.FileIO + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.fileio.html + - transform: AvroIO + description: PTransforms for reading from and writing to [Avro](https://avro.apache.org/) files. + implementations: + - language: java + name: org.apache.beam.sdk.io.AvroIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/AvroIO.html + - language: py + name: apache_beam.io.avroio + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.avroio.html + - language: go + name: github.com/apache/beam/sdks/go/pkg/beam/io/avroio + url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/avroio + - transform: TextIO + description: PTransforms for reading and writing text files. + implementations: + - language: java + name: org.apache.beam.sdk.io.TextIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html + - language: py + name: apache_beam.io.textio + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html + - language: go + name: github.com/apache/beam/sdks/go/pkg/beam/io/textio + url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/textio + - transform: TFRecordIO + description: PTransforms for reading and writing [TensorFlow TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) files. + implementations: + - language: java + name: org.apache.beam.sdk.io.TFRecordIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TFRecordIO.html + - language: py + name: apache_beam.io.tfrecordio + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.tfrecordio.html + - transform: XmlIO + description: Transforms for reading and writing XML files using [JAXB](https://www.oracle.com/technical-resources/articles/javase/jaxb.html) mappers. + implementations: + - language: java + name: org.apache.beam.sdk.io.xml.XmlIO + url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/xml/XmlIO.html + - transform: TikaIO + description: Transforms for parsing arbitrary files using [Apache Tika](https://tika.apache.org/). + implementations: + - language: java + name: org.apache.beam.sdk.io.tika.TikaIO + url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/tika/TikaIO.html + - transform: ParquetIO + description: IO for reading from and writing to [Parquet](https://parquet.apache.org/) files. + docs: /documentation/io/built-in/parquet/ + implementations: + - language: java + name: org.apache.beam.sdk.io.parquet.ParquetIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/parquet/ParquetIO.html + - language: py + name: apache_beam.io.parquetio + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.parquetio.html + - transform: ThriftIO + description: PTransforms for reading and writing files containing [Thrift](https://thrift.apache.org/)-encoded data. + implementations: + - language: java + name: org.apache.beam.sdk.io.thrift.ThriftIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/thrift/ThriftIO.html + - transform: VcfIO + description: A source for reading from [VCF files](https://samtools.github.io/hts-specs/VCFv4.2.pdf) (version 4.x). + implementations: + - language: py + name: apache_beam.io.vcfio + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.vcfio.html + - transform: S3IO + description: A source for reading from and writing to [Amazon S3](https://aws.amazon.com/s3/). + implementations: + - language: py + name: apache_beam.io.aws.s3io + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.aws.s3io.html + - transform: GcsIO + description: A source for reading from and writing to [Google Cloud Storage](https://cloud.google.com/storage). + implementations: + - language: py + name: apache_beam.io.gcp.gcsio + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsio.html + - name: FileSystem + description: Beam provides a File system interface that defines APIs for writing file systems agnostic code. Several I/O connectors are implemented as a FileSystem implementation. + rows: + - transform: HadoopFileSystem + description: "`FileSystem` implementation for accessing [Hadoop](https://hadoop.apache.org/) Distributed File System files." + implementations: + - language: java + name: org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrar + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.html + - language: py + name: apache_beam.io.hadoopfilesystem + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.hadoopfilesystem.html + - transform: GcsFileSystem + description: "`FileSystem` implementation for [Google Cloud Storage](https://cloud.google.com/storage)." + implementations: + - language: java + name: org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.html + - language: py + name: apache_beam.io.gcp.gcsfilesystem + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsfilesystem.html + - language: go + name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs + url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs + - transform: LocalFileSystem + description: "`FileSystem` implementation for accessing files on disk." + implementations: + - language: java + name: org.apache.beam.sdk.io.LocalFileSystemRegistrar + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/LocalFileSystemRegistrar.html + - language: py + name: apache_beam.io.localfilesystem + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.localfilesystem.html + - language: go + name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local + url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local + - transform: S3FileSystem + description: "`FileSystem` implementation for [Amazon S3](https://aws.amazon.com/s3/)." + implementations: + - language: java + name: org.apache.beam.sdk.io.aws.s3.S3FileSystemRegistrar + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/package-summary.html + - transform: In-memory + description: "`FileSystem` implementation in memory; useful for testing." + implementations: + - language: go + name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs + url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs + - name: Messaging + description: These I/O connectors typically involve working with unbounded sources that come from messaging sources. + rows: + - transform: KinesisIO + description: PTransforms for reading from and writing to [Kinesis](https://aws.amazon.com/kinesis/) streams. + implementations: + - language: java + name: org.apache.beam.sdk.io.kinesis.KinesisIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kinesis/KinesisIO.html + - transform: AmqpIO + description: AMQP 1.0 protocol using the Apache QPid Proton-J library + implementations: + - language: java + name: org.apache.beam.sdk.io.amqp.AmqpIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/amqp/AmqpIO.html + - transform: KafkaIO + description: Read and Write PTransforms for [Apache Kafka](https://kafka.apache.org/). + implementations: + - language: java + name: org.apache.beam.sdk.io.kafka.KafkaIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kafka/KafkaIO.html + - language: py + name: apache_beam.io.external.kafka + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.kafka.html + - transform: PubSubIO + description: Read and Write PTransforms for [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) streams. + implementations: + - language: java + name: org.apache.beam.sdk.io.gcp.pubsub.PubsubIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html + - language: py + name: apache_beam.io.gcp.pubsub + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.pubsub.html + - language: py + name: apache_beam.io.external.gcp.pubsub + url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.gcp.pubsub.html + - language: go + name: github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio + url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio + - transform: JmsIO + description: An unbounded source for [JMS](https://www.oracle.com/java/technologies/java-message-service.html) destinations (queues or topics). + implementations: + - language: java + name: org.apache.beam.sdk.io.jms.JmsIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jms/JmsIO.html + - transform: MqttIO + description: An unbounded source for [MQTT](https://mqtt.org/) broker. + implementations: + - language: java + name: org.apache.beam.sdk.io.mqtt.MqttIO + url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/mqtt/MqttIO.html + - transform: RabbitMqIO + description: A IO to publish or consume messages with a RabbitMQ broker. + implementations: + - language: java + name: org.apache.beam.sdk.io.rabbitmq.RabbitMqIO + url: https://github.com/apache/beam/blob/master/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java Review comment: @epicfaace could you file JIRA issues for these? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 437309) Time Spent: 40m (was: 0.5h) > Update IOs documentation links > ------------------------------ > > Key: BEAM-9916 > URL: https://issues.apache.org/jira/browse/BEAM-9916 > Project: Beam > Issue Type: Improvement > Components: website > Reporter: Alexey Romanenko > Assignee: Ashwin Ramaswami > Priority: P3 > Time Spent: 40m > Remaining Estimate: 0h > > Currently, on the webpage > [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to > their code on github, which could be quite odd for users. After a > [discussion|https://lists.apache.org/thread.html/r2c7bb9a2e13f1b8c296930fc8a36edafe731f49b5182567989afcb98%40%3Cdev.beam.apache.org%3E] > on dev@, it was decided to link them to the latest doc reference (like > [https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html] > for FileIO). > -- This message was sent by Atlassian Jira (v8.3.4#803005)