[jira] [Assigned] (HIVE-20165) Enable ZLIB for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-20165: -- Assignee: Prasanth Jayachandran (was: Gopal V) > Enable ZLIB for streaming ingest > > > Key: HIVE-20165 > URL: https://issues.apache.org/jira/browse/HIVE-20165 > Project: Hive > Issue Type: Bug > Components: Streaming, Transactions >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20165.1.branch-3.patch, HIVE-20165.1.patch, > HIVE-20165.2.patch, HIVE-20165.3.patch > > > Per [~gopalv]'s recommendation tried running streaming ingest with and > without zlib. Following are the numbers > > *Compression: NONE* > Total rows committed: 9380 > Throughput: *156* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *14.1 G* /apps/hive/warehouse/prasanth.db/culvert > > *Compression: ZLIB* > Total rows committed: 9210 > Throughput: *1535000* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *7.4 G* /apps/hive/warehouse/prasanth.db/culvert > > ZLIB is getting us 2x compression and only 2% lesser throughput. We should > enable ZLIB by default for streaming ingest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20165) Enable ZLIB for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-20165: -- Assignee: Gopal V (was: Prasanth Jayachandran) > Enable ZLIB for streaming ingest > > > Key: HIVE-20165 > URL: https://issues.apache.org/jira/browse/HIVE-20165 > Project: Hive > Issue Type: Bug > Components: Streaming, Transactions >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Gopal V >Priority: Major > Attachments: HIVE-20165.1.branch-3.patch, HIVE-20165.1.patch, > HIVE-20165.2.patch, HIVE-20165.3.patch > > > Per [~gopalv]'s recommendation tried running streaming ingest with and > without zlib. Following are the numbers > > *Compression: NONE* > Total rows committed: 9380 > Throughput: *156* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *14.1 G* /apps/hive/warehouse/prasanth.db/culvert > > *Compression: ZLIB* > Total rows committed: 9210 > Throughput: *1535000* rows/second > $ hdfs dfs -du -s -h /apps/hive/warehouse/prasanth.db/culvert > *7.4 G* /apps/hive/warehouse/prasanth.db/culvert > > ZLIB is getting us 2x compression and only 2% lesser throughput. We should > enable ZLIB by default for streaming ingest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20165) Enable ZLIB for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-20165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-20165: > Enable ZLIB for streaming ingest > > > Key: HIVE-20165 > URL: https://issues.apache.org/jira/browse/HIVE-20165 > Project: Hive > Issue Type: Bug > Components: Streaming, Transactions >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > Per [~gopalv]'s recommendation tried running streaming ingest with and > without zlib. Following are the numbers > > *Compression: NONE* > Total rows committed: 9380 > Throughput: *156* rows/second > [prasanth@cn105-10 culvert]$ hdfs dfs -du -s -h > /apps/hive/warehouse/prasanth.db/culvert > *14.1 G* /apps/hive/warehouse/prasanth.db/culvert > > *Compression: ZLIB* > Total rows committed: 9210 > Throughput: *1535000* rows/second > [prasanth@cn105-10 culvert]$ hdfs dfs -du -s -h > /apps/hive/warehouse/prasanth.db/culvert > *7.4 G* /apps/hive/warehouse/prasanth.db/culvert > > ZLIB is getting us 2x compression and only 2% lesser throughput. We should > enable ZLIB by default for streaming ingest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
