[ 
https://issues.apache.org/jira/browse/PARQUET-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17147587#comment-17147587
 ] 

ASF GitHub Bot commented on PARQUET-1643:
-----------------------------------------

samarthjain commented on pull request #671:
URL: https://github.com/apache/parquet-mr/pull/671#issuecomment-650971182


   @dbtsai 
   > Since airlift is pure Java implementation, what's the performance 
implications for zstd? I saw there is a benchmark for GZIP, but I don't see 
benchmark for other codecs.
   It looks like the zstd Airlift implementation doesn't implement the Hadoop 
APIs. It can be integrated within Parquet, but will take some work worth 
definitely worthy of another PR.
    
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Use airlift non-native implementations for GZIP, LZ0 and LZ4 codecs
> -------------------------------------------------------------------
>
>                 Key: PARQUET-1643
>                 URL: https://issues.apache.org/jira/browse/PARQUET-1643
>             Project: Parquet
>          Issue Type: Improvement
>            Reporter: Samarth Jain
>            Assignee: Samarth Jain
>            Priority: Major
>              Labels: pull-request-available
>
> [~rdblue] pointed me to [https://github.com/airlift/aircompressor] which 
> provides non-native implementations of compression codecs. It claims to be 
> much faster than native wrappers that parquet uses. This Jira is to track the 
> work needed for exploring using these codecs, getting benchmark results and 
> making changes including not needing to pool compressors and decompressors 
> anymore. Note that this doesn't include SNAPPY since Parquet already has its 
> own non-hadoopy implementation for it. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to