[
https://issues.apache.org/jira/browse/IMPALA-11431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878057#comment-17878057
]
Csaba Ringhofer commented on IMPALA-11431:
------------------------------------------
I see 2 issues here:
1.
test_compute_stats_with_structs itself is problematic, as it has side effects.
It calls COMPUTE STATS for functional_*.complextypes_structs and
complextypes_nested_structs, which modifies the state of tables read by many
tests. While this is is not the cause of the issue, it is something to clean up.
2.
HMS throws an exception in some cases with "Column row__id doesn't exist in
table"
row__id is related to full ACID table (when the test runs for ORC the table is
full ACID)
This leads to keeping the old empty stats and failing the test. I saw 140 error
like this during the exhaustive test run, so this also effects some other
tables. It is not clear why is this sporadic though.
Impala adds a "synthetic" row__id column to full ACID tables, so these columns
don't come from HMS and we should try to read their statistics.
The full exception in HMS:
{code:java}
2024-08-29T02:35:24,264 ERROR [TThreadPoolServer WorkerProcess-142]
metastore.ObjectStore: Error retrieving statistics via jdo
org.apache.hadoop.hive.metastore.api.MetaException: Column row__id doesn't
exist in table complextypes_structs in database functional_orc_def
at
org.apache.hadoop.hive.metastore.ObjectStore.validateTableCols(ObjectStore.java:10342)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at
org.apache.hadoop.hive.metastore.ObjectStore.getMTableColumnStatistics(ObjectStore.java:10277)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at
org.apache.hadoop.hive.metastore.ObjectStore.access$3100(ObjectStore.java:295)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at
org.apache.hadoop.hive.metastore.ObjectStore$20.getJdoResult(ObjectStore.java:10434)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at
org.apache.hadoop.hive.metastore.ObjectStore$20.getJdoResult(ObjectStore.java:10426)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:4345)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at
org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:10454)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at
org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:10412)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_422]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_422]
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at com.sun.proxy.$Proxy33.getTableColumnStatistics(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_statistics_req(HiveMetaStore.java:7186)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_422]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_422]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:160)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:121)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at com.sun.proxy.$Proxy34.get_table_statistics_req(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_statistics_req.getResult(ThriftHiveMetastore.java:22613)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_statistics_req.getResult(ThriftHiveMetastore.java:22592)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
~[libthrift-0.16.0.jar:0.16.0]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_422]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_422]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
~[hadoop-common-3.1.1.7.2.18.0-641.jar:?]
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
~[hive-standalone-metastore-3.1.3000.2024.0.19.0-170.jar:3.1.3000.2024.0.19.0-170]
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
~[libthrift-0.16.0.jar:0.16.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_422]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_422]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_422] {code}
> TestComputeStatsWithNestedTypes.test_compute_stats_with_structs fails in an
> exhaustive build
> --------------------------------------------------------------------------------------------
>
> Key: IMPALA-11431
> URL: https://issues.apache.org/jira/browse/IMPALA-11431
> Project: IMPALA
> Issue Type: Bug
> Reporter: Daniel Becker
> Assignee: Daniel Becker
> Priority: Blocker
> Labels: broken-build
>
> In one of the exhaustive builds,
> query_test.test_nested_types.TestComputeStatsWithNestedTypes.test_compute_stats_with_structs
> fails:
> {code:java}
> query_test/test_nested_types.py:252: in test_compute_stats_with_structs
> self.run_test_case('QueryTest/compute-stats-with-structs', vector)
> common/impala_test_suite.py:778: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:588: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:469: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E
> 'alltypes','STRUCT<ti:TINYINT,si:SMALLINT,i:INT,bi:BIGINT,b:BOOLEAN,f:FLOAT,do:DOUBLE,da:DATE,ts:TIMESTAMP,s1:STRING,s2:STRING,c1:CHAR(1),c2:CHAR(3),vc:VARCHAR(10),de1:DECIMAL(5,0),de2:DECIMAL(10,3)>',-1,-1,-1,-1.0,-1,-1
> ==
> 'alltypes','STRUCT<ti:TINYINT,si:SMALLINT,i:INT,bi:BIGINT,b:BOOLEAN,f:FLOAT,do:DOUBLE,da:DATE,ts:TIMESTAMP,s1:STRING,s2:STRING,c1:CHAR(1),c2:CHAR(3),vc:VARCHAR(10),de1:DECIMAL(5,0),de2:DECIMAL(10,3)>',-1,-1,-1,-1,-1,-1
> E 'id','INT',6,0,4,4.0,-1,-1 != 'id','INT',-1,-1,4,4,-1,-1
> E 'small_struct','STRUCT<i:INT,s:STRING>',-1,-1,-1,-1.0,-1,-1 ==
> 'small_struct','STRUCT<i:INT,s:STRING>',-1,-1,-1,-1,-1,-1
> E 'str','STRING',6,0,11,10.3333330154,-1,-1 !=
> 'str','STRING',-1,-1,-1,-1,-1,-1
> E 'tiny_struct','STRUCT<b:BOOLEAN>',-1,-1,-1,-1.0,-1,-1 ==
> 'tiny_struct','STRUCT<b:BOOLEAN>',-1,-1,-1,-1,-1,-1
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
