[
https://issues.apache.org/jira/browse/BEAM-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15547423#comment-15547423
]
ASF GitHub Bot commented on BEAM-570:
-------------------------------------
GitHub user katsiapis opened a pull request:
https://github.com/apache/incubator-beam/pull/1053
[BEAM-570] Title of the pull request
- Getting rid of CompressionTypes.ZLIB and CompressionTypes.NO_COMPRESSION.
- Introducing BZIP2 compression in analogy to Dataflow Java's BZIP2,
towards resolution of https://issues.apache.org/jira/browse/BEAM-570.
- Introducing SNAPPY codec support for AVRO conciseness and in order to
fully resolve https://issues.apache.org/jira/browse/BEAM-570.
- Moving avroio from compression_type to codec as per various discussions.
- A few cleanups in avroio.
- Making textio more DRY and doing a few cleanups.
- Raising exceptions when splitting is requested for compressed source
since that should never happen (guaranteed by the service for the supported
compression types).
- Using cStringIO instead of StringIO in various places as decided in some
other discussions.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/katsiapis/incubator-beam bz2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/1053.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1053
----
commit bd44e76b80e4edf4f922b9a26f7b359c4ede2008
Author: Gus Katsiapis <[email protected]>
Date: 2016-10-05T02:41:07Z
Several enhancements to Dataflow (part 2 of 2).
----
> Update AvroSource to support more compression types
> ---------------------------------------------------
>
> Key: BEAM-570
> URL: https://issues.apache.org/jira/browse/BEAM-570
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py
> Reporter: Chamikara Jayalath
> Assignee: Chamikara Jayalath
>
> Python AvroSource [1] currently only support 'deflate' compression. We should
> update it to support other compression types supported by the Avro library
> (e.g.: snappy, bzip2).
> [1]
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/avroio.py
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
