[jira] [Commented] (HIVE-10226) Column stats for Date columns not supported
[
https://issues.apache.org/jira/browse/HIVE-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14487396#comment-14487396
]
Hive QA commented on HIVE-10226:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12723972/HIVE-10226.4.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 8666 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3344/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3344/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3344/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12723972 - PreCommit-HIVE-TRUNK-Build
Column stats for Date columns not supported
---
Key: HIVE-10226
URL: https://issues.apache.org/jira/browse/HIVE-10226
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-10226.1.patch, HIVE-10226.2.patch,
HIVE-10226.3.patch, HIVE-10226.4.patch
{noformat}
hive explain analyze table revenues compute statistics for columns;
2015-03-30 23:47:45,133 ERROR [main()]: ql.Driver
(SessionState.java:printError(951)) - FAILED: UDFArgumentTypeException Only
integer/long/timestamp/float/double/string/binary/boolean/decimal type
argument is accepted but date is passed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10226) Column stats for Date columns not supported
[
https://issues.apache.org/jira/browse/HIVE-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14487503#comment-14487503
]
Swarnim Kulkarni commented on HIVE-10226:
-
+1. Looks great. Thanks for making the changes.
Column stats for Date columns not supported
---
Key: HIVE-10226
URL: https://issues.apache.org/jira/browse/HIVE-10226
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-10226.1.patch, HIVE-10226.2.patch,
HIVE-10226.3.patch, HIVE-10226.4.patch
{noformat}
hive explain analyze table revenues compute statistics for columns;
2015-03-30 23:47:45,133 ERROR [main()]: ql.Driver
(SessionState.java:printError(951)) - FAILED: UDFArgumentTypeException Only
integer/long/timestamp/float/double/string/binary/boolean/decimal type
argument is accepted but date is passed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10226) Column stats for Date columns not supported
[
https://issues.apache.org/jira/browse/HIVE-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484791#comment-14484791
]
Swarnim Kulkarni commented on HIVE-10226:
-
Few very minor comments on RB. +1 otherwise.
Column stats for Date columns not supported
---
Key: HIVE-10226
URL: https://issues.apache.org/jira/browse/HIVE-10226
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-10226.1.patch, HIVE-10226.2.patch,
HIVE-10226.3.patch
{noformat}
hive explain analyze table revenues compute statistics for columns;
2015-03-30 23:47:45,133 ERROR [main()]: ql.Driver
(SessionState.java:printError(951)) - FAILED: UDFArgumentTypeException Only
integer/long/timestamp/float/double/string/binary/boolean/decimal type
argument is accepted but date is passed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10226) Column stats for Date columns not supported
[
https://issues.apache.org/jira/browse/HIVE-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485640#comment-14485640
]
Jason Dere commented on HIVE-10226:
---
test failures do not look related
Column stats for Date columns not supported
---
Key: HIVE-10226
URL: https://issues.apache.org/jira/browse/HIVE-10226
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-10226.1.patch, HIVE-10226.2.patch,
HIVE-10226.3.patch, HIVE-10226.4.patch
{noformat}
hive explain analyze table revenues compute statistics for columns;
2015-03-30 23:47:45,133 ERROR [main()]: ql.Driver
(SessionState.java:printError(951)) - FAILED: UDFArgumentTypeException Only
integer/long/timestamp/float/double/string/binary/boolean/decimal type
argument is accepted but date is passed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10226) Column stats for Date columns not supported
[
https://issues.apache.org/jira/browse/HIVE-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482225#comment-14482225
]
Ashutosh Chauhan commented on HIVE-10226:
-
Using long column in metastore table to store max and min date is fine, but
reusing {{LongColumnStatsData}} in thrift interface to return result not so
much. I think we should add new thrift structure {DateColumnStatsData}}
something like :
{code}
struct DateColumnStatsData {
1: optional Date minDate,
2: optional Date maxDate,
3: required i64 numNulls,
4: required i64 numDVs
}
{code}
and than metastore after retrieving # of days from backend, should compute
Date, populate this struct and return results back to client.
Column stats for Date columns not supported
---
Key: HIVE-10226
URL: https://issues.apache.org/jira/browse/HIVE-10226
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-10226.1.patch
{noformat}
hive explain analyze table revenues compute statistics for columns;
2015-03-30 23:47:45,133 ERROR [main()]: ql.Driver
(SessionState.java:printError(951)) - FAILED: UDFArgumentTypeException Only
integer/long/timestamp/float/double/string/binary/boolean/decimal type
argument is accepted but date is passed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10226) Column stats for Date columns not supported
[
https://issues.apache.org/jira/browse/HIVE-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482299#comment-14482299
]
Jason Dere commented on HIVE-10226:
---
[~ashutoshc] I'll see what I can do here.
[~gopalv] Is there a more complete stack trace available here, in case this
issue still crops up after I make these changes? The qfile test (which uses
analyze table) seemed to work ok.
Column stats for Date columns not supported
---
Key: HIVE-10226
URL: https://issues.apache.org/jira/browse/HIVE-10226
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-10226.1.patch
{noformat}
hive explain analyze table revenues compute statistics for columns;
2015-03-30 23:47:45,133 ERROR [main()]: ql.Driver
(SessionState.java:printError(951)) - FAILED: UDFArgumentTypeException Only
integer/long/timestamp/float/double/string/binary/boolean/decimal type
argument is accepted but date is passed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10226) Column stats for Date columns not supported
[
https://issues.apache.org/jira/browse/HIVE-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482321#comment-14482321
]
Gopal V commented on HIVE-10226:
[~jdere]: I am running the latest build against a hive-1.0 metastore, that
won't be ever hit via the qtests.
{code}
org.apache.thrift.protocol.TProtocolException: Cannot write a TUnion with no
set value!
at org.apache.thrift.TUnion$TUnionStandardScheme.write(TUnion.java:240)
at org.apache.thrift.TUnion$TUnionStandardScheme.write(TUnion.java:213)
at org.apache.thrift.TUnion.write(TUnion.java:152)
at
org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj$ColumnStatisticsObjStandardScheme.write(ColumnStatisticsObj.java:550)
at
org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj$ColumnStatisticsObjStandardScheme.write(ColumnStatisticsObj.java:488)
at
org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj.write(ColumnStatisticsObj.java:414)
at
org.apache.hadoop.hive.metastore.api.TableStatsResult$TableStatsResultStandardScheme.write(TableStatsResult.java:388)
at
org.apache.hadoop.hive.metastore.api.TableStatsResult$TableStatsResultStandardScheme.write(TableStatsResult.java:338)
at
org.apache.hadoop.hive.metastore.api.TableStatsResult.write(TableStatsResult.java:288)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_statistics_req_result$get_table_statistics_req_resultStandardScheme.write(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_statistics_req_result$get_table_statistics_req_resultStandardScheme.write(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_statistics_req_result.write(ThriftHiveMetastore.java)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:53)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
The error is probably not relevant except for rolling upgrade scenarios (the
standard one is client upgraded, server yet to be restarted), since the error
is not a FATAL.
Adding a new Stats type is probably a non-backwards compat change anyway, so
perhaps we can mark this as a wire protocol change.
Column stats for Date columns not supported
---
Key: HIVE-10226
URL: https://issues.apache.org/jira/browse/HIVE-10226
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-10226.1.patch
{noformat}
hive explain analyze table revenues compute statistics for columns;
2015-03-30 23:47:45,133 ERROR [main()]: ql.Driver
(SessionState.java:printError(951)) - FAILED: UDFArgumentTypeException Only
integer/long/timestamp/float/double/string/binary/boolean/decimal type
argument is accepted but date is passed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10226) Column stats for Date columns not supported
[
https://issues.apache.org/jira/browse/HIVE-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482310#comment-14482310
]
Hive QA commented on HIVE-10226:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12723434/HIVE-10226.1.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 8653 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-bucket5.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-parallel_orderby.q-reduce_deduplicate.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-infer_bucket_sort_bucketed_table.q-bucket4.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-smb_mapjoin_8.q - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3299/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3299/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3299/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12723434 - PreCommit-HIVE-TRUNK-Build
Column stats for Date columns not supported
---
Key: HIVE-10226
URL: https://issues.apache.org/jira/browse/HIVE-10226
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-10226.1.patch
{noformat}
hive explain analyze table revenues compute statistics for columns;
2015-03-30 23:47:45,133 ERROR [main()]: ql.Driver
(SessionState.java:printError(951)) - FAILED: UDFArgumentTypeException Only
integer/long/timestamp/float/double/string/binary/boolean/decimal type
argument is accepted but date is passed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10226) Column stats for Date columns not supported
[
https://issues.apache.org/jira/browse/HIVE-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482557#comment-14482557
]
Swarnim Kulkarni commented on HIVE-10226:
-
[~jdere] Left minor comments on the review but +1 to Ashutosh's suggestion.
That should provide us additional flexibility in future(ex: locales etc).
Column stats for Date columns not supported
---
Key: HIVE-10226
URL: https://issues.apache.org/jira/browse/HIVE-10226
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-10226.1.patch
{noformat}
hive explain analyze table revenues compute statistics for columns;
2015-03-30 23:47:45,133 ERROR [main()]: ql.Driver
(SessionState.java:printError(951)) - FAILED: UDFArgumentTypeException Only
integer/long/timestamp/float/double/string/binary/boolean/decimal type
argument is accepted but date is passed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
