[jira] [Commented] (HIVE-14169) Honor --incremental flag only if TableOutputFormat is used
[
https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384925#comment-15384925
]
Sahil Takiar commented on HIVE-14169:
-
Hey [~taoli-hwx]
* Yes, by default it is still false
* For non-table formats we came to the conclusion that there is no real benefit
to using BufferedRows. It only really makes sense if the table output format is
used. The reason is that if table output format is used along with
BufferedRows, then the BufferedRows can calculate the optimal sizing for each
row that it prints out. However, this isn't applicable for non-table formats.
This is why I made the change to stop honoring the value of incremental if a
non-table format is used.
Also, I am going to close this JIRA and mark it as a duplicate of HIVE-14170 -
since it doesn't make sense to commit these changes without HIVE-14170 along
with it.
> Honor --incremental flag only if TableOutputFormat is used
> --
>
> Key: HIVE-14169
> URL: https://issues.apache.org/jira/browse/HIVE-14169
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14169.1.patch
>
>
> * When Beeline prints out a {{ResultSet}} to stdout it uses the
> {{BeeLine.print}} method
> * This method takes the {{ResultSet}} from the completed query and uses a
> specified {{OutputFormat}} to print the rows (by default it uses
> {{TableOutputFormat}})
> * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class
> (either a {{IncrementalRows}} or a {{BufferedRows}} class)
> The advantage of {{BufferedRows}} is that it can do a global calculation of
> the column width, however, this is only useful for {{TableOutputFormat}}. So
> there is no need to buffer all the rows if a different {{OutputFormat}} is
> used. This JIRA will change the behavior of the {{--incremental}} flag so
> that it is only honored if {{TableOutputFormat}} is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14169) Honor --incremental flag only if TableOutputFormat is used
[
https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383363#comment-15383363
]
Tao Li commented on HIVE-14169:
---
Hi Sahil,
2 quick questions:
1. I think the default setting for "--incremental" is still false with your
change, right? If that's true, we still go into the code path of buffered rows
if the "--incremental" is not specified, which I think is a typical use case.
2. Looks like we always do the incremental rows regardless of the setting for
"--incremental" for non-table formats. What if the user specifies a non-table
format and also "--incremental=false"? Do we want to do buffered rows in this
case?
Thanks.
> Honor --incremental flag only if TableOutputFormat is used
> --
>
> Key: HIVE-14169
> URL: https://issues.apache.org/jira/browse/HIVE-14169
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14169.1.patch
>
>
> * When Beeline prints out a {{ResultSet}} to stdout it uses the
> {{BeeLine.print}} method
> * This method takes the {{ResultSet}} from the completed query and uses a
> specified {{OutputFormat}} to print the rows (by default it uses
> {{TableOutputFormat}})
> * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class
> (either a {{IncrementalRows}} or a {{BufferedRows}} class)
> The advantage of {{BufferedRows}} is that it can do a global calculation of
> the column width, however, this is only useful for {{TableOutputFormat}}. So
> there is no need to buffer all the rows if a different {{OutputFormat}} is
> used. This JIRA will change the behavior of the {{--incremental}} flag so
> that it is only honored if {{TableOutputFormat}} is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14169) Honor --incremental flag only if TableOutputFormat is used
[
https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382902#comment-15382902
]
Sahil Takiar commented on HIVE-14169:
-
[~thejas] any comments on this JIRA or HIVE-14170?
Thanks!
--Sahil
> Honor --incremental flag only if TableOutputFormat is used
> --
>
> Key: HIVE-14169
> URL: https://issues.apache.org/jira/browse/HIVE-14169
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14169.1.patch
>
>
> * When Beeline prints out a {{ResultSet}} to stdout it uses the
> {{BeeLine.print}} method
> * This method takes the {{ResultSet}} from the completed query and uses a
> specified {{OutputFormat}} to print the rows (by default it uses
> {{TableOutputFormat}})
> * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class
> (either a {{IncrementalRows}} or a {{BufferedRows}} class)
> The advantage of {{BufferedRows}} is that it can do a global calculation of
> the column width, however, this is only useful for {{TableOutputFormat}}. So
> there is no need to buffer all the rows if a different {{OutputFormat}} is
> used. This JIRA will change the behavior of the {{--incremental}} flag so
> that it is only honored if {{TableOutputFormat}} is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14169) Honor --incremental flag only if TableOutputFormat is used
[
https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371136#comment-15371136
]
Sahil Takiar commented on HIVE-14169:
-
Test failures look unrelated, [~thejas] and [~spena] can you review?
> Honor --incremental flag only if TableOutputFormat is used
> --
>
> Key: HIVE-14169
> URL: https://issues.apache.org/jira/browse/HIVE-14169
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14169.1.patch
>
>
> * When Beeline prints out a {{ResultSet}} to stdout it uses the
> {{BeeLine.print}} method
> * This method takes the {{ResultSet}} from the completed query and uses a
> specified {{OutputFormat}} to print the rows (by default it uses
> {{TableOutputFormat}})
> * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class
> (either a {{IncrementalRows}} or a {{BufferedRows}} class)
> The advantage of {{BufferedRows}} is that it can do a global calculation of
> the column width, however, this is only useful for {{TableOutputFormat}}. So
> there is no need to buffer all the rows if a different {{OutputFormat}} is
> used. This JIRA will change the behavior of the {{--incremental}} flag so
> that it is only honored if {{TableOutputFormat}} is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14169) Honor --incremental flag only if TableOutputFormat is used
[
https://issues.apache.org/jira/browse/HIVE-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369277#comment-15369277
]
Hive QA commented on HIVE-14169:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816907/HIVE-14169.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10296 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/444/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/444/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-444/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12816907 - PreCommit-HIVE-MASTER-Build
> Honor --incremental flag only if TableOutputFormat is used
> --
>
> Key: HIVE-14169
> URL: https://issues.apache.org/jira/browse/HIVE-14169
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14169.1.patch
>
>
> * When Beeline prints out a {{ResultSet}} to stdout it uses the
> {{BeeLine.print}} method
> * This method takes the {{ResultSet}} from the completed query and uses a
> specified {{OutputFormat}} to print the rows (by default it uses
> {{TableOutputFormat}})
> * The {{print}} method also wraps the {{ResultSet}} into a {{Rows}} class
> (either a {{IncrementalRows}} or a {{BufferedRows}} class)
> The advantage of {{BufferedRows}} is that it can do a global calculation of
> the column width, however, this is only useful for {{TableOutputFormat}}. So
> there is no need to buffer all the rows if a different {{OutputFormat}} is
> used. This JIRA will change the behavior of the {{--incremental}} flag so
> that it is only honored if {{TableOutputFormat}} is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
