[jira] [Commented] (HIVE-2266) Fix compression parameters
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504479#comment-13504479 ] Harsh J commented on HIVE-2266: --- bq. Hadoop loads native compression libraries. I believe that they are platform dependent hence I do not assume that they always have same compression ratio. Please correct me if I am wrong here. Compression is based on standard algorithms, which is platform independent. The native code is platform-dependent cause of the library references it has. > Fix compression parameters > -- > > Key: HIVE-2266 > URL: https://issues.apache.org/jira/browse/HIVE-2266 > Project: Hive > Issue Type: Bug >Reporter: Vaibhav Aggarwal >Assignee: Vaibhav Aggarwal > Attachments: HIVE-2266-2.patch, HIVE-2266.patch > > > There are a number of places where compression values are not set correctly > in FileSinkOperator. This results in uncompressed files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2266) Fix compression parameters
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13095747#comment-13095747 ] Vaibhav Aggarwal commented on HIVE-2266: This patch attempts to fix a bug in the existing functionality in two ways: 1. In HiveFileFormatUtils.java, wrong jobconf is getting passed which is clear from the context. 2. In other cases the compression parameters are not getting set. The only difference this patch produces from the current behavior is smaller file sizes on file system. I am not sure how to write a hive query which can verify difference in file sizes. Do you have any ideas which can help me add some quick tests for this? The current test executes though the code checking that it does not result in any Exception or Error. It does not compare file size. > Really? Which platforms are you talking about? Can you tell me how to > reproduce this interesting behavior? Hadoop loads native compression libraries. I believe that they are platform dependent hence I do not assume that they always have same compression ratio. Please correct me if I am wrong here. In any case I think this is a broken existing functionality in Hive which we should fix. > Fix compression parameters > -- > > Key: HIVE-2266 > URL: https://issues.apache.org/jira/browse/HIVE-2266 > Project: Hive > Issue Type: Bug >Reporter: Vaibhav Aggarwal >Assignee: Vaibhav Aggarwal > Attachments: HIVE-2266-2.patch, HIVE-2266.patch > > > There are a number of places where compression values are not set correctly > in FileSinkOperator. This results in uncompressed files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2266) Fix compression parameters
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092161#comment-13092161 ] Vaibhav Aggarwal commented on HIVE-2266: Hi Carl The patch fixes compression which means that there is a difference in file size only. Besides the file size will differ on different platforms. Hence there is no way to test this except for manual test. This test case executes the new code and makes sure that it works. Sent from my phone. - Reply message - From: "Carl Steinbach (JIRA)" Date: Fri, Aug 26, 2011 6:13 pm Subject: [jira] [Updated] (HIVE-2266) Fix compression parameters To: "Aggarwal, Vaibhav" [ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2266: - Status: Open (was: Patch Available) @Vaibhav: The compress.q test passes even without your patch in place. Can you please supply a testcase that fails without your patch? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira > Fix compression parameters > -- > > Key: HIVE-2266 > URL: https://issues.apache.org/jira/browse/HIVE-2266 > Project: Hive > Issue Type: Bug >Reporter: Vaibhav Aggarwal >Assignee: Vaibhav Aggarwal > Attachments: HIVE-2266-2.patch, HIVE-2266.patch > > > There are a number of places where compression values are not set correctly > in FileSinkOperator. This results in uncompressed files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2266) Fix compression parameters
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072549#comment-13072549 ] Vaibhav Aggarwal commented on HIVE-2266: Carl, I have attached the patch. > Fix compression parameters > -- > > Key: HIVE-2266 > URL: https://issues.apache.org/jira/browse/HIVE-2266 > Project: Hive > Issue Type: Bug >Reporter: Vaibhav Aggarwal >Assignee: Vaibhav Aggarwal > Attachments: HIVE-2266.patch > > > There are a number of places where compression values are not set correctly > in FileSinkOperator. This results in uncompressed files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2266) Fix compression parameters
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072546#comment-13072546 ] Carl Steinbach commented on HIVE-2266: -- @Vaibav: Is this patch ready for review? If so, please attach the patch to this ticket and click "Submit Patch". Thanks. > Fix compression parameters > -- > > Key: HIVE-2266 > URL: https://issues.apache.org/jira/browse/HIVE-2266 > Project: Hive > Issue Type: Bug >Reporter: Vaibhav Aggarwal >Assignee: Vaibhav Aggarwal > > There are a number of places where compression values are not set correctly > in FileSinkOperator. This results in uncompressed files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2266) Fix compression parameters
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060778#comment-13060778 ] Vaibhav Aggarwal commented on HIVE-2266: Review request: https://reviews.apache.org/r/1017/ > Fix compression parameters > -- > > Key: HIVE-2266 > URL: https://issues.apache.org/jira/browse/HIVE-2266 > Project: Hive > Issue Type: Bug >Reporter: Vaibhav Aggarwal >Assignee: Vaibhav Aggarwal > > There are a number of places where compression values are not set correctly > in FileSinkOperator. This results in uncompressed files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2266) Fix compression parameters
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060779#comment-13060779 ] jirapos...@reviews.apache.org commented on HIVE-2266: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1017/ --- Review request for hive. Summary --- Fix compression parameters for FileSinkOperator This addresses bug HIVE-2266. https://issues.apache.org/jira/browse/HIVE-2266 Diffs - ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 404c4d3 ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java 092a5cd ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 8fe5984 Diff: https://reviews.apache.org/r/1017/diff Testing --- Thanks, Vaibhav > Fix compression parameters > -- > > Key: HIVE-2266 > URL: https://issues.apache.org/jira/browse/HIVE-2266 > Project: Hive > Issue Type: Bug >Reporter: Vaibhav Aggarwal >Assignee: Vaibhav Aggarwal > > There are a number of places where compression values are not set correctly > in FileSinkOperator. This results in uncompressed files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
