Hello Holden,
Thank you for the response, but what is "splittable gzip"?
Best, Oliver
On Tue, Dec 6, 2022 at 9:22 AM Holden Karau <hol...@pigscanfly.ca> wrote:
> There is the splittable gzip Hadoop input format, maybe someone could
> extend that to use support bgzip?
>
> On Tue, Dec 6, 2022 at 1:43 PM Oliver Ruebenacker <
> oliv...@broadinstitute.org> wrote:
>
>>
>> Hello Chris,
>>
>> Yes, you can use gunzip/gzip to uncompress a file created by bgzip, but
>> to start reading from somewhere other than the beginning of the file, you
>> would need to use an index to tell you where the blocks start. Originally,
>> a Tabix index was used and is still the popular choice, although other
>> types of indices also exist.
>>
>> Best, Oliver
>>
>> On Mon, Dec 5, 2022 at 6:17 PM Chris Nauroth <cnaur...@apache.org> wrote:
>>
>>> Sorry, I misread that in the original email.
>>>
>>> This is my first time looking at bgzip. I see from the documentation
>>> that it is putting some additional framing around gzip and producing a
>>> series of small blocks, such that you can create an index of the file and
>>> decompress individual blocks instead of the whole file. That's interesting,
>>> because it could potentially support a splittable format. (Plain gzip isn't
>>> splittable.)
>>>
>>> I also noticed that it states it is "compatible with" gzip. I tried a
>>> basic test of running bgzip on a file, which produced a .gz output file,
>>> and then running the same spark.read.text code sample from earlier. Sure
>>> enough, I was able to read the data. This implies there is at least some
>>> basic compatibility, so that you could read files created by bgzip.
>>> However, that read would not be optimized in any way to take advantage of
>>> an index file. There also would not be any way to produce bgzip-style
>>> output like in the df.write.option code sample. To achieve either of those,
>>> it would require writing a custom Hadoop compression codec to integrate
>>> more closely with the data format.
>>>
>>> Chris Nauroth
>>>
>>>
>>> On Mon, Dec 5, 2022 at 2:08 PM Oliver Ruebenacker <
>>> oliv...@broadinstitute.org> wrote:
>>>
>>>>
>>>> Hello,
>>>>
>>>> Thanks for the response, but I mean compressed with bgzip
>>>> <http://www.htslib.org/doc/bgzip.html>, not bzip2.
>>>>
>>>> Best, Oliver
>>>>
>>>> On Fri, Dec 2, 2022 at 4:44 PM Chris Nauroth <cnaur...@apache.org>
>>>> wrote:
>>>>
>>>>> Hello Oliver,
>>>>>
>>>>> Yes, Spark makes this possible using the Hadoop compression codecs and
>>>>> the Hadoop-compatible FileSystem interface [1]. Here is an example of
>>>>> reading:
>>>>>
>>>>> df = spark.read.text("gs://<GCS bucket>/data/shakespeare-bz2")
>>>>> df.show(10)
>>>>>
>>>>> This is using a test data set of the complete works of Shakespeare,
>>>>> stored as text and compressed to a single .bz2 file. This code sample
>>>>> didn't need to do anything special to declare that it's working with bzip2
>>>>> compression, because the Hadoop compression codecs detect that the file
>>>>> has
>>>>> a .bz2 extension and automatically assume it needs to be decompressed
>>>>> before presenting it to our code in the DataFrame as text.
>>>>>
>>>>> On the write side, if you wanted to declare a particular kind of
>>>>> output compression, you can do it with a write option like this:
>>>>>
>>>>> df.write.option("compression",
>>>>> "org.apache.hadoop.io.compress.BZip2Codec").text("gs://<GCS
>>>>> bucket>/data/shakespeare-bz2-copy")
>>>>>
>>>>> This writes the contents of the DataFrame, stored as text and
>>>>> compressed to .bz2 files in the destination path.
>>>>>
>>>>> My example is testing with a GCS bucket (scheme "gs:"), but you can
>>>>> also switch the Hadoop file system interface to target other file systems
>>>>> like S3 (scheme "s3a:"). Hadoop maintains documentation on how to
>>>>> configure
>>>>> the S3AFIleSystem, including how to pass credentials for access to the S3
>>>>> bucket [2].
>>>>>
>>>>> Note that for big data use cases, other compression codecs like Snappy
>>>>> are generally preferred for greater efficiency. (Of course, we're not
>>>>> always in complete control of the data formats we're given, so the support
>>>>> for bz2 is there.)
>>>>>
>>>>> [1]
>>>>> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/index.html
>>>>> [2]
>>>>> https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html
>>>>>
>>>>> Chris Nauroth
>>>>>
>>>>>
>>>>> On Fri, Dec 2, 2022 at 11:32 AM Oliver Ruebenacker <
>>>>> oliv...@broadinstitute.org> wrote:
>>>>>
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Is it possible to read/write a DataFrame from/to a set of bgzipped
>>>>>> files? Can it read from/write to AWS S3? Thanks!
>>>>>>
>>>>>> Best, Oliver
>>>>>>
>>>>>> --
>>>>>> Oliver Ruebenacker, Ph.D. (he)
>>>>>> Senior Software Engineer, Knowledge Portal Network
>>>>>> <http://kp4cd.org/>, Flannick Lab <http://www.flannicklab.org/>, Broad
>>>>>> Institute <http://www.broadinstitute.org/>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Oliver Ruebenacker, Ph.D. (he)
>>>> Senior Software Engineer, Knowledge Portal Network <http://kp4cd.org/>,
>>>> Flannick Lab <http://www.flannicklab.org/>, Broad Institute
>>>> <http://www.broadinstitute.org/>
>>>>
>>>
>>
>> --
>> Oliver Ruebenacker, Ph.D. (he)
>> Senior Software Engineer, Knowledge Portal Network <http://kp4cd.org/>,
>> Flannick
>> Lab <http://www.flannicklab.org/>, Broad Institute
>> <http://www.broadinstitute.org/>
>>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
--
Oliver Ruebenacker, Ph.D. (he)
Senior Software Engineer, Knowledge Portal Network
<http://kp4cd.org/>, Flannick
Lab <http://www.flannicklab.org/>, Broad Institute
<http://www.broadinstitute.org/>
