[jira] [Assigned] (IMPALA-10251) test_decimal_queries and test_tpcds_queries may run out of memory on ASAN builds
[
https://issues.apache.org/jira/browse/IMPALA-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-10251:
--
Assignee: (was: Tim Armstrong)
> test_decimal_queries and test_tpcds_queries may run out of memory on ASAN
> builds
>
>
> Key: IMPALA-10251
> URL: https://issues.apache.org/jira/browse/IMPALA-10251
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Priority: Critical
> Labels: broken-build, flaky
>
> Encountered these test failures due to running out of memory:
> {code:java}
> query_test.test_decimal_queries.TestDecimalQueries.test_queries[protocol:
> beeswax | exec_option: {'disable_codegen_rows_threshold': 0,
> 'disable_codegen': 'false', 'decimal_v2': 'true', 'batch_size': 1} |
> table_format:
> orc/def/block]query_test.test_tpcds_queries.TestTpcdsQuery.test_tpcds_q51a[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]query_test.test_decimal_queries.TestDecimalQueries.test_queries[protocol:
> hs2 | exec_option: {'disable_codegen_rows_threshold': 0, 'disable_codegen':
> 'true', 'decimal_v2': 'true', 'batch_size': 1} | table_format:
> parquet/none]query_test.test_tpcds_queries.TestTpcdsDecimalV2Query.test_tpcds_q18a[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]query_test.test_tpcds_queries.TestTpcdsDecimalV2Query.test_tpcds_q23_1[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]query_test.test_tpcds_queries.TestTpcdsDecimalV2Query.test_tpcds_q4[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none] {code}
> The error is
> {code}
> query_test/test_decimal_queries.py:55: in test_queries
> self.run_test_case('QueryTest/decimal', vector)
> common/impala_test_suite.py:662: in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:600: in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:909: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:365: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:386: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:Failed to get minimum memory reservation of 68.01 MB on
> daemon impala-ec2-centos74-r5-4xlarge-ondemand-109b.vpc.cloudera.com:27001
> for query 2b41e9326825c48d:15a4d931 due to following error: Failed to
> increase reservation by 68.01 MB because it would exceed the applicable
> reservation limit for the "Process" ReservationTracker:
> reservation_limit=10.20 GB reservation=10.16 GB used_reservation=0
> child_reservations=10.16 GB
> E The top 5 queries that allocated memory under this tracker are:
> E Query(ca4c47a813876f70:d626ca0c): Reservation=9.60 GB
> ReservationLimit=9.60 GB OtherMemory=115.84 MB Total=9.71 GB Peak=9.71 GB
> E Query(3f4e984e6b54204d:2597d87f): Reservation=426.77 MB
> ReservationLimit=9.60 GB OtherMemory=113.19 MB Total=539.95 MB Peak=540.00 MB
> E Query(6c4aca5cba7141b7:cb0d9a40): Limit=512.00 MB
> Reservation=68.03 MB ReservationLimit=409.60 MB OtherMemory=2.59 MB
> Total=70.62 MB Peak=71.93 MB
> E Query(e84c2ac37bac9309:7eddf8bb): Reservation=14.94 MB
> ReservationLimit=9.60 GB OtherMemory=5.55 MB Total=20.48 MB Peak=39.46 MB
> E Query(eb44550f4e6aebbd:4b72c396): Reservation=12.31 MB
> ReservationLimit=9.60 GB OtherMemory=5.85 MB Total=18.16 MB Peak=18.16 MB
> E
> E
> E
> E
> E
> E Memory is likely oversubscribed. Reducing query concurrency or
> configuring admission control may help avoid this error.
> {code}
--
This message was sent by Atlassian Jira
[jira] [Assigned] (IMPALA-10344) Error loading functional_orc_def.alltypesaggmultifiles in hive
[
https://issues.apache.org/jira/browse/IMPALA-10344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-10344:
--
Assignee: (was: Tim Armstrong)
> Error loading functional_orc_def.alltypesaggmultifiles in hive
> --
>
> Key: IMPALA-10344
> URL: https://issues.apache.org/jira/browse/IMPALA-10344
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 4.0
>Reporter: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
> Attachments: hive-metastore.out, hive-server2.out,
> load-functional-query-exhaustive-hive-generated-orc-def-block.sql.log
>
>
> Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
> Exception when loading 11 partitions in table alltypesaggmultifiles
> {noformat}
> 0: jdbc:hive2://localhost:11050/default> insert into table
> functional_orc_def.al
> ltypesaggmultifiles partition (year, month, day) SELECT id, bool_col,
> tinyint_co
> l, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col,
> st
> ring_col, timestamp_col, year, month, day FROM
> functional.alltypesaggmultifiles
> where id % 4 = 1;
> going to print operations logs
> printed operations logs
> going to print operations logs
> INFO : Compiling
> command(queryId=jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9):
> insert into table functional_orc_def.alltypesaggmultifiles partition (year,
> month, day) SELECT id, bool_col, tinyint_col, smallint_col, int_col,
> bigint_col, float_col, double_col, date_string_col, string_col,
> timestamp_col, year, month, day FROM functional.alltypesaggmultifiles where
> id % 4 = 1
> INFO : No Stats for functional@alltypesaggmultifiles, Columns: float_col,
> int_col, string_col, bool_col, date_string_col, smallint_col, timestamp_col,
> tinyint_col, bigint_col, id, double_col
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:id,
> type:int, comment:null), FieldSchema(name:bool_col, type:boolean,
> comment:null), FieldSchema(name:tinyint_col, type:tinyint, comment:null),
> FieldSchema(name:smallint_col, type:smallint, comment:null),
> FieldSchema(name:int_col, type:int, comment:null),
> FieldSchema(name:bigint_col, type:bigint, comment:null),
> FieldSchema(name:float_col, type:float, comment:null),
> FieldSchema(name:double_col, type:double, comment:null),
> FieldSchema(name:date_string_col, type:string, comment:null),
> FieldSchema(name:string_col, type:string, comment:null),
> FieldSchema(name:timestamp_col, type:timestamp, comment:null),
> FieldSchema(name:year, type:int, comment:null), FieldSchema(name:month,
> type:int, comment:null), FieldSchema(name:day, type:int, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9);
> Time taken: 0.036 seconds
> INFO : Executing
> command(queryId=jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9):
> insert into table functional_orc_def.alltypesaggmultifiles partition (year,
> month, day) SELECT id, bool_col, tinyint_col, smallint_col, int_col,
> bigint_col, float_col, double_col, date_string_col, string_col,
> timestamp_col, year, month, day FROM functional.alltypesaggmultifiles where
> id % 4 = 1
> INFO : Query ID = jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9
> INFO : Total jobs = 1
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-1:MAPRED] in serial mode
> INFO : Subscribed to counters: [] for queryId:
> jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9
> INFO : Session is already open
> INFO : Dag name: insert into table functional_orc_def.all...1 (Stage-1)
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1605304119768_0012)
> printed operations logs
> [2K--
> [2K[36;1mVERTICES MODESTATUS TOTAL COMPLETED
> RUNNING PENDING FAILED KILLED
> [22;0m[2K--
> [2KMap 1 .. container SUCCEEDED 42 420
> 0 0 0
> Reducer 2container RUNNING 1 010
> 0 0
> [2K--
> [2K[31;1mVERTICES: 01/02 [=>>-] 97% ELAPSED
> TIME: 1.46 s
> [22;0m[2K--
>
[jira] [Resolved] (IMPALA-10235) Averaged timer profile counters can be negative for trivial queries
[
https://issues.apache.org/jira/browse/IMPALA-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10235.
Fix Version/s: Impala 4.0
Assignee: Tim Armstrong
Resolution: Fixed
This was introduced by part 1 of IMPALA-9382 and fixed by this part of the
below patch.
@@ -1842,7 +2137,8 @@ RuntimeProfileBase::AveragedCounter::GetStats() const {
if (!vals.empty()) {
sort(vals.begin(), vals.end());
result.num_vals = vals.size();
-result.mean = std::accumulate(vals.begin(), vals.end(), 0) /
result.num_vals;
+T sum = std::accumulate(vals.begin(), vals.end(), 0L);
+result.mean = sum / result.num_vals;
result.min = vals[0];
result.max = vals.back();
int end_idx = vals.size() - 1;
@@ -1902,6 +2198,50 @@ void
RuntimeProfileBase::AveragedCounter::PrettyPrintImpl(
commit 9429bd779de986d3e61858bef7e258bd73a2cacd
Author: Tim Armstrong
Date: Sun May 17 16:37:46 2020 -0700
IMPALA-9382: part 2/3: aggregate profiles sent to coordinator
This reworks the status reporting so that serialized
AggregatedRuntimeProfile objects are sent from executors
to coordinators. These profiles are substantially denser
and faster to process for higher mt_dop values. The aggregation
is also done in a single step, merging the aggregated thrift
profile from the executor directly into the final aggregated
profile, instead of converting it to an unaggregated profile
first.
The changes required were:
* A new Update() method for AggregatedRuntimeProfile that
updates the profile from a serialised AggregateRuntimeProfile
for a subset of the instances. The code is generalized from the
existing InitFromThrift() code path.
* Per-fragment reports included in the status report protobuf
when --gen_experimental_profile=true.
* Logic on the coordinator that either consumes serialized
AggregatedRuntimeProfile per fragment, when
--gen_experimental_profile=true, or consumes a serialized
RuntimeProfile per finstance otherwise.
This also adds support for event sequences and time series
in the aggregated profile, so the amount of information
in the aggregated profile is now on par with the basic profile.
We also finish off support for JSON profile. The JSON profile is
more stripped down because we do not need to round-trip profiles
via JSON and it is a much less dense profile representation.
Part 3 will clean up and improve the display of the profile.
Testing:
* Add sanity tests for aggregated runtime profile.
* Add unit tests to exercise aggregation of the various counter types
* Ran core tests.
Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca
Reviewed-on: http://gerrit.cloudera.org:8080/16057
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Averaged timer profile counters can be negative for trivial queries
> ---
>
> Key: IMPALA-10235
> URL: https://issues.apache.org/jira/browse/IMPALA-10235
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Tim Armstrong
>Priority: Major
> Labels: newbie, ramp-up
> Fix For: Impala 4.0
>
> Attachments: profile-output.txt
>
>
> Steps to reproduce on master:
> {code:java}
> stakiar @ stakiar-desktop -bash ~/Impala 2020-10-13 11:13:02 master
> [74] → ./bin/impala-shell.sh -q "select sleep(100) from functional.alltypes
> limit 25" -p > profile-output.txt
> ...
> Query: select sleep(100) from functional.alltypes limit 25
> Query submitted at: 2020-10-13 11:13:07 (Coordinator:
> http://stakiar-desktop:25000)
> Query progress can be monitored at:
> http://stakiar-desktop:25000/query_plan?query_id=694f94671571d4d1:cdec9db9
> Fetched 25 row(s) in 2.64s
> {code}
> Attached the contents of {{profile-output.txt}}
> Relevant portion of the profile:
> {code:java}
> Averaged Fragment F00:(Total: 2s603ms, non-child: 272.519us, % non-child:
> 0.01%)
> ...
>- CompletionTime: -1665218428.000ns
> ...
>- TotalThreadsTotalWallClockTime: -1686005515.000ns
> - TotalThreadsSysTime: 0.000ns
> - TotalThreadsUserTime: 2.151ms
> ...
>- TotalTime: -1691524485.000ns
> {code}
> For whatever reason, this only affects the averaged fragment profile. For
> this query, there was only one coordinator fragment and thus only one
> fragment instance. The coordinator fragment instance showed normal timer
> values:
> {code:java}
> Coordinator Fragment F00:
> ...
> - CompletionTime: 2s629ms
> ...
> - TotalThreadsTotalWallClockTime: 2s608ms
>- TotalThreadsSysTime: 0.000ns
>-
[jira] [Resolved] (IMPALA-6870) SummaryStatsCounter should be included in averaged profile
[ https://issues.apache.org/jira/browse/IMPALA-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-6870. --- Fix Version/s: Impala 4.0 Resolution: Fixed commit 9429bd779de986d3e61858bef7e258bd73a2cacd Author: Tim Armstrong Date: Sun May 17 16:37:46 2020 -0700 IMPALA-9382: part 2/3: aggregate profiles sent to coordinator This reworks the status reporting so that serialized AggregatedRuntimeProfile objects are sent from executors to coordinators. These profiles are substantially denser and faster to process for higher mt_dop values. The aggregation is also done in a single step, merging the aggregated thrift profile from the executor directly into the final aggregated profile, instead of converting it to an unaggregated profile first. The changes required were: * A new Update() method for AggregatedRuntimeProfile that updates the profile from a serialised AggregateRuntimeProfile for a subset of the instances. The code is generalized from the existing InitFromThrift() code path. * Per-fragment reports included in the status report protobuf when --gen_experimental_profile=true. * Logic on the coordinator that either consumes serialized AggregatedRuntimeProfile per fragment, when --gen_experimental_profile=true, or consumes a serialized RuntimeProfile per finstance otherwise. This also adds support for event sequences and time series in the aggregated profile, so the amount of information in the aggregated profile is now on par with the basic profile. We also finish off support for JSON profile. The JSON profile is more stripped down because we do not need to round-trip profiles via JSON and it is a much less dense profile representation. Part 3 will clean up and improve the display of the profile. Testing: * Add sanity tests for aggregated runtime profile. * Add unit tests to exercise aggregation of the various counter types * Ran core tests. Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca Reviewed-on: http://gerrit.cloudera.org:8080/16057 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > SummaryStatsCounter should be included in averaged profile > -- > > Key: IMPALA-6870 > URL: https://issues.apache.org/jira/browse/IMPALA-6870 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.12.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: observability > Fix For: Impala 4.0 > > > Summary stats like FooterProcessingTime don't show up in the averaged > fragment. We should be able to merge these counters to produce overall > statistics: -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10377) Improve the accuracy of resource estimation
[ https://issues.apache.org/jira/browse/IMPALA-10377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-10377: --- Target Version: Impala 4.0 > Improve the accuracy of resource estimation > --- > > Key: IMPALA-10377 > URL: https://issues.apache.org/jira/browse/IMPALA-10377 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 3.4.0 >Reporter: liuyao >Assignee: liuyao >Priority: Major > Labels: estimate, memory, statistics > Original Estimate: 120h > Remaining Estimate: 120h > > PlanNode does not consider some factors when estimating memory, this will > cause a large error rate > > AggregationNode > > 1.The memory occupied by hash table's own data structure is not considered. > Hash table inserts a new value, which will add a bucket. The size of a bucket > is 16 bytes. > 2.When estimating the NDV of merge aggregation, if there are multiple > grouping exprs, it may be divided by the number of Fragment Instances several > times, and it should be divided only once. > 3.When estimating the NDV of merge aggregation, and there are multiple > grouping exprs, the estimated memory is much smaller than the actual use. > 4.If there is no grouping exprs, the estimated memory is much larger than the > actual use. > 5.If the NDV of grouping exprs is very small, the estimated memory is much > larger than the actual use. > > SortNode > 1.Estimate the memory usage of external sort. the estimated memory is much > smaller than the actual use. > > > HashJoinNode > 1.The memory occupied by hash table's own data structure is not > considered.Hash Table will keep duplicate data, so the size of DuplicateNode > should be considered. > 2.Hash table will create multiple buckets in advance. The size of these > buckets should be considered. > > KuduScanNode > 1.Estimate memory by scanning all columns,the estimated memory is much larger > than the actual use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10156) test_unmatched_schema flaky with wrong results
[
https://issues.apache.org/jira/browse/IMPALA-10156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10156.
Fix Version/s: Impala 4.0
Resolution: Fixed
> test_unmatched_schema flaky with wrong results
> --
>
> Key: IMPALA-10156
> URL: https://issues.apache.org/jira/browse/IMPALA-10156
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
> Fix For: Impala 4.0
>
>
> {noformat}
> Error Message
> query_test/test_scanners.py:244: in test_unmatched_schema
> self.run_test_case('QueryTest/test-unmatched-schema', vector)
> common/impala_test_suite.py:693: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:529: in __verify_results_and_errors
> replace_filenames_with_placeholder) common/test_result_verifier.py:456: in
> verify_raw_results VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results E assert Comparing
> QueryTestResults (expected vs actual): E 1001,'Name1',94611,5000 ==
> 1001,'Name1',94611,5000 E 1002,'Name2',94611,5000 !=
> 1001,'Name1',94611,5000 E 1003,'Name3',94611,5000 !=
> 1002,'Name2',94611,5000 E 1004,'Name4',94611,5000 !=
> 1002,'Name2',94611,5000 E 1005,'Name5',94611,5000 !=
> 1003,'Name3',94611,5000 E 1006,'Name16',94612,15000 !=
> 1003,'Name3',94611,5000 E 1006,'Name16',94612,5000 !=
> 1004,'Name4',94611,5000 E 1006,'Name16',94616,15000 !=
> 1004,'Name4',94611,5000 E 1006,'Name16',94616,5000 !=
> 1005,'Name5',94611,5000 E 1006,'Name6',94616,15000 !=
> 1005,'Name5',94611,5000 E 1006,'Name6',94616,5000 !=
> 1006,'Name16',94612,15000 E 1106,'Name16',94612,15000 !=
> 1006,'Name16',94612,15000 E 1106,'Name16',94612,5000 !=
> 1006,'Name16',94612,5000 E 1106,'Name16',94616,15000 !=
> 1006,'Name16',94612,5000 E 1106,'Name16',94616,5000 !=
> 1006,'Name16',94616,15000 E 1106,'Name6',94612,15000 !=
> 1006,'Name16',94616,15000 E 1106,'Name6',94612,5000 !=
> 1006,'Name16',94616,5000 E 1106,'Name6',94616,15000 !=
> 1006,'Name16',94616,5000 E 1106,'Name6',94616,5000 !=
> 1006,'Name6',94616,15000 E None != 1006,'Name6',94616,15000 E None !=
> 1006,'Name6',94616,5000 E None != 1006,'Name6',94616,5000 E None !=
> 1106,'Name16',94612,15000 E None != 1106,'Name16',94612,15000 E None
> != 1106,'Name16',94612,5000 E None != 1106,'Name16',94612,5000 E None
> != 1106,'Name16',94616,15000 E None != 1106,'Name16',94616,15000 E
> None != 1106,'Name16',94616,5000 E None != 1106,'Name16',94616,5000 E
> None != 1106,'Name6',94612,15000 E None != 1106,'Name6',94612,15000 E
> None != 1106,'Name6',94612,5000 E None != 1106,'Name6',94612,5000 E
> None != 1106,'Name6',94616,15000 E None != 1106,'Name6',94616,15000 E
> None != 1106,'Name6',94616,5000 E None != 1106,'Name6',94616,5000 E
> Number of rows returned (expected vs actual): 19 != 38
> Stacktrace
> query_test/test_scanners.py:244: in test_unmatched_schema
> self.run_test_case('QueryTest/test-unmatched-schema', vector)
> common/impala_test_suite.py:693: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:529: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 1001,'Name1',94611,5000 == 1001,'Name1',94611,5000
> E 1002,'Name2',94611,5000 != 1001,'Name1',94611,5000
> E 1003,'Name3',94611,5000 != 1002,'Name2',94611,5000
> E 1004,'Name4',94611,5000 != 1002,'Name2',94611,5000
> E 1005,'Name5',94611,5000 != 1003,'Name3',94611,5000
> E 1006,'Name16',94612,15000 != 1003,'Name3',94611,5000
> E 1006,'Name16',94612,5000 != 1004,'Name4',94611,5000
> E 1006,'Name16',94616,15000 != 1004,'Name4',94611,5000
> E 1006,'Name16',94616,5000 != 1005,'Name5',94611,5000
> E 1006,'Name6',94616,15000 != 1005,'Name5',94611,5000
> E 1006,'Name6',94616,5000 != 1006,'Name16',94612,15000
> E 1106,'Name16',94612,15000 != 1006,'Name16',94612,15000
> E 1106,'Name16',94612,5000 != 1006,'Name16',94612,5000
> E 1106,'Name16',94616,15000 != 1006,'Name16',94612,5000
> E 1106,'Name16',94616,5000 !=
[jira] [Updated] (IMPALA-10377) Improve the accuracy of resource estimation
[ https://issues.apache.org/jira/browse/IMPALA-10377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-10377: --- Fix Version/s: (was: Impala 4.0) > Improve the accuracy of resource estimation > --- > > Key: IMPALA-10377 > URL: https://issues.apache.org/jira/browse/IMPALA-10377 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 3.4.0 >Reporter: liuyao >Assignee: liuyao >Priority: Major > Labels: estimate, memory, statistics > Original Estimate: 120h > Remaining Estimate: 120h > > PlanNode does not consider some factors when estimating memory, this will > cause a large error rate > > AggregationNode > > 1.The memory occupied by hash table's own data structure is not considered. > Hash table inserts a new value, which will add a bucket. The size of a bucket > is 16 bytes. > 2.When estimating the NDV of merge aggregation, if there are multiple > grouping exprs, it may be divided by the number of Fragment Instances several > times, and it should be divided only once. > 3.When estimating the NDV of merge aggregation, and there are multiple > grouping exprs, the estimated memory is much smaller than the actual use. > 4.If there is no grouping exprs, the estimated memory is much larger than the > actual use. > 5.If the NDV of grouping exprs is very small, the estimated memory is much > larger than the actual use. > > SortNode > 1.Estimate the memory usage of external sort. the estimated memory is much > smaller than the actual use. > > > HashJoinNode > 1.The memory occupied by hash table's own data structure is not > considered.Hash Table will keep duplicate data, so the size of DuplicateNode > should be considered. > 2.Hash table will create multiple buckets in advance. The size of these > buckets should be considered. > > KuduScanNode > 1.Estimate memory by scanning all columns,the estimated memory is much larger > than the actual use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10370) Thrift Message Incompatibility Detector
[ https://issues.apache.org/jira/browse/IMPALA-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244104#comment-17244104 ] Tim Armstrong commented on IMPALA-10370: [~zagol] you can exclude BackendGflags.thrift, CatalogInternalService.thrift, CatalogService.thrift, DataSinks.thrift, Data.thrift, Descriptors.thrift, Exprs.thrift, Frontend.thrift, ImpalaInternalService.thrift, JniCatalog.thrift, Logging.thrift, NetworkTest.thrift, Partitions.thrift, Planner.thrift, PlanNodes.thrift, ResourceProfile.thrift, Results.thrift, SqlConstraints.thrift, StatestoreService.thrift, Zip.thrift Those should only be used internally. Some files like Types.thrift have definitions that are used both internally and externally, but I think if you exclude the above files it will avoid most false positives. > Thrift Message Incompatibility Detector > --- > > Key: IMPALA-10370 > URL: https://issues.apache.org/jira/browse/IMPALA-10370 > Project: IMPALA > Issue Type: Improvement >Reporter: junwen yang >Priority: Major > Attachments: impala_thrift_incompatibility.txt > > > Regarding the issue IMPALA-8243 caused by the incompatibility of thrift > message, we have created a static checker which keeps track of the thrift > file change, and detect potential incompatibility: > # Add/delete required field. The thrift guidelines suggests _Any new fields > that you add should be optional_. > # The tag number of a field has been changed. Also, the thrift guidelines > suggests _Don’t change the numeric tags for any existing fields_. > # A required field has been changed to optional, or an optional field has > been changed to required. According to the guidelines , _*Required Is > Forever* You should be very careful about marking fields as required. If at > some point you wish to stop writing or sending a required field, it will be > problematic to change the field to an optional field — old readers will > consider messages without this field to be incomplete and may reject or drop > them unintentionally. You should consider writing application-specific custom > validation routines for your buffers instead. Some have come to the > conclusion that using required does more harm than good; they prefer to use > only optional. However, this view is not universal._ > We have applied our checker on the frequently maintained IMPALA versions: > refs/tags/2.10.0, refs/tags/2.11.0, refs/tags/2.12.0, refs/tags/2.7.0, > refs/tags/2.8.0, refs/tags/2.9.0, refs/tags/3.0.0, refs/tags/3.0.1, > refs/tags/3.1.0, refs/tags/3.2.0, refs/tags/3.3.0, refs/tags/3.4.0, we found > more than 1000 problems as attached. > The results reported by our checker got confirmed by developers of HBASE and > our checker is requested by them, which can be found at HBASE-25340. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10377) Improve the accuracy of resource estimation
[ https://issues.apache.org/jira/browse/IMPALA-10377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244097#comment-17244097 ] Tim Armstrong commented on IMPALA-10377: [~liuyao] thanks for investigating all these issues, it would be great to make these more accurate. We need to be a bit careful if the estimates change a lot because it can disrupt production workloads. Look forward to seeing a patch. > Improve the accuracy of resource estimation > --- > > Key: IMPALA-10377 > URL: https://issues.apache.org/jira/browse/IMPALA-10377 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 3.4.0 >Reporter: liuyao >Assignee: liuyao >Priority: Major > Labels: estimate, memory, statistics > Fix For: Impala 4.0 > > Original Estimate: 120h > Remaining Estimate: 120h > > PlanNode does not consider some factors when estimating memory, this will > cause a large error rate > > AggregationNode > > 1.The memory occupied by hash table's own data structure is not considered. > Hash table inserts a new value, which will add a bucket. The size of a bucket > is 16 bytes. > 2.When estimating the NDV of merge aggregation, if there are multiple > grouping exprs, it may be divided by the number of Fragment Instances several > times, and it should be divided only once. > 3.When estimating the NDV of merge aggregation, and there are multiple > grouping exprs, the estimated memory is much smaller than the actual use. > 4.If there is no grouping exprs, the estimated memory is much larger than the > actual use. > 5.If the NDV of grouping exprs is very small, the estimated memory is much > larger than the actual use. > > SortNode > 1.Estimate the memory usage of external sort. the estimated memory is much > smaller than the actual use. > > > HashJoinNode > 1.The memory occupied by hash table's own data structure is not > considered.Hash Table will keep duplicate data, so the size of DuplicateNode > should be considered. > 2.Hash table will create multiple buckets in advance. The size of these > buckets should be considered. > > KuduScanNode > 1.Estimate memory by scanning all columns,the estimated memory is much larger > than the actual use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10377) Improve the accuracy of resource estimation
[ https://issues.apache.org/jira/browse/IMPALA-10377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244093#comment-17244093 ] Tim Armstrong commented on IMPALA-10377: I had some ideas about improving accuracy of estimates for the preaggregations, I'll link those here. > Improve the accuracy of resource estimation > --- > > Key: IMPALA-10377 > URL: https://issues.apache.org/jira/browse/IMPALA-10377 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 3.4.0 >Reporter: liuyao >Assignee: liuyao >Priority: Major > Labels: estimate, memory, statistics > Fix For: Impala 4.0 > > Original Estimate: 120h > Remaining Estimate: 120h > > PlanNode does not consider some factors when estimating memory, this will > cause a large error rate > > AggregationNode > > 1.The memory occupied by hash table's own data structure is not considered. > Hash table inserts a new value, which will add a bucket. The size of a bucket > is 16 bytes. > 2.When estimating the NDV of merge aggregation, if there are multiple > grouping exprs, it may be divided by the number of Fragment Instances several > times, and it should be divided only once. > 3.When estimating the NDV of merge aggregation, and there are multiple > grouping exprs, the estimated memory is much smaller than the actual use. > 4.If there is no grouping exprs, the estimated memory is much larger than the > actual use. > 5.If the NDV of grouping exprs is very small, the estimated memory is much > larger than the actual use. > > SortNode > 1.Estimate the memory usage of external sort. the estimated memory is much > smaller than the actual use. > > > HashJoinNode > 1.The memory occupied by hash table's own data structure is not > considered.Hash Table will keep duplicate data, so the size of DuplicateNode > should be considered. > 2.Hash table will create multiple buckets in advance. The size of these > buckets should be considered. > > KuduScanNode > 1.Estimate memory by scanning all columns,the estimated memory is much larger > than the actual use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10371) test_java_udfs crash impalad if result spooling is enabled
[
https://issues.apache.org/jira/browse/IMPALA-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243726#comment-17243726
]
Tim Armstrong commented on IMPALA-10371:
I think that must be it! I'm concerned now that we might have other instances
of this issue though - we don't have a documented or enforced invariant that
the evaluator must be opened on the same thread as it's evaluated on.
I.e. this seems like more of a bug in HiveUdfCall than in the caller.
> test_java_udfs crash impalad if result spooling is enabled
> --
>
> Key: IMPALA-10371
> URL: https://issues.apache.org/jira/browse/IMPALA-10371
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.4.0
>Reporter: Riza Suminto
>Priority: Major
> Attachments: 46a19881-resolved.txt, hs_err_pid12878.log
>
>
> The following test query from TestUdfExecution::test_java_udfs crash impalad
> when result spooling is enabled.
> {code:java}
> select throws_exception() from functional.alltypestiny{code}
> The following is a truncated JVM crash log related to the crash
> {code:java}
> --- T H R E A D ---Current thread
> (0x0fb4c000): JavaThread "Thread-700" [_thread_in_native, id=30853,
> stack(0x7f79715ff000,0x7f7971dff000)]Stack:
> [0x7f79715ff000,0x7f7971dff000], sp=0x7f7971dfa280, free
> space=8172k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0xb6b032]
> V [libjvm.so+0x4f14bd]
> V [libjvm.so+0x80fa8f]
> V [libjvm.so+0x7e0991]
> V [libjvm.so+0x69fa10]
> j
> org.apache.impala.TestUdfException.evaluate()Lorg/apache/hadoop/io/BooleanWritable;+9
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6af9ba]
> V [libjvm.so+0xa1def8]
> V [libjvm.so+0xa1f8d5]
> V [libjvm.so+0x7610f8] JVM_InvokeMethod+0x128
> J 2286
> sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (0 bytes) @ 0x7f7acb553ced [0x7f7acb553c00+0xed]
> J 6921 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7f7acbd1de38 [0x7f7acbd1ddc0+0x78]
> J 3645 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7f7acaf6e894 [0x7f7acaf6e640+0x254]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6af9ba]
> V [libjvm.so+0x72c046]
> V [libjvm.so+0x730523]
> C 0x7f7ab4c5d0d0
> C [impalad+0x26a2648]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x7a
> C [impalad+0x26a25cb]
> impala::ScalarExprEvaluator::GetValue(impala::TupleRow const*)+0x2b
> C [impalad+0x21f4f78]
> impala::AsciiQueryResultSet::AddRows(std::vector std::allocator > const&, impala::RowBatch*,
> int, int)+0x4c2
> C [impalad+0x25c5862]
> impala::BufferedPlanRootSink::GetNext(impala::RuntimeState*,
> impala::QueryResultSet*, int, bool*, long)+0x70c
> C [impalad+0x296cf17] impala::Coordinator::GetNext(impala::QueryResultSet*,
> int, bool*, long)+0x557
> C [impalad+0x219f5fe] impala::ClientRequestState::FetchRowsInternal(int,
> impala::QueryResultSet*, long)+0x6b2
> C [impalad+0x219d98e] impala::ClientRequestState::FetchRows(int,
> impala::QueryResultSet*, long)+0x46
> C [impalad+0x21c1d29]
> impala::ImpalaServer::FetchInternal(impala::TUniqueId, bool, int,
> beeswax::Results*)+0x717
> C [impalad+0x21bbde9] impala::ImpalaServer::fetch(beeswax::Results&,
> beeswax::QueryHandle const&, bool, int)+0x577
> {code}
> If result spooling is enabled, BufferedPlanRootSink will be used and
> ScalarExprEvaluation will be called in BufferedPlanRootSink::GetNext, leading
> to this crash.
> Without result spooling, BlockingPlanRootSink will be used and
> ScalarExprEvaluation is called in BlockingPlanRootSink::Send. No crash happen
> when result spooling is disabled.
> Attached is the full JVM crash log and resolved minidump.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10371) test_java_udfs crash impalad if result spooling is enabled
[
https://issues.apache.org/jira/browse/IMPALA-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242797#comment-17242797
]
Tim Armstrong commented on IMPALA-10371:
[~rizaon] I believe it should be OK for it to be evaluated by any of the query
execution threads. ScalarExprEvaluator is not thread-safe though, so possibly
if it's being accessed by multiple threads concurrently, then there could be a
bug.
> test_java_udfs crash impalad if result spooling is enabled
> --
>
> Key: IMPALA-10371
> URL: https://issues.apache.org/jira/browse/IMPALA-10371
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.4.0
>Reporter: Riza Suminto
>Priority: Major
> Attachments: 46a19881-resolved.txt, hs_err_pid12878.log
>
>
> The following test query from TestUdfExecution::test_java_udfs crash impalad
> when result spooling is enabled.
> {code:java}
> select throws_exception() from functional.alltypestiny{code}
> The following is a truncated JVM crash log related to the crash
> {code:java}
> --- T H R E A D ---Current thread
> (0x0fb4c000): JavaThread "Thread-700" [_thread_in_native, id=30853,
> stack(0x7f79715ff000,0x7f7971dff000)]Stack:
> [0x7f79715ff000,0x7f7971dff000], sp=0x7f7971dfa280, free
> space=8172k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0xb6b032]
> V [libjvm.so+0x4f14bd]
> V [libjvm.so+0x80fa8f]
> V [libjvm.so+0x7e0991]
> V [libjvm.so+0x69fa10]
> j
> org.apache.impala.TestUdfException.evaluate()Lorg/apache/hadoop/io/BooleanWritable;+9
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6af9ba]
> V [libjvm.so+0xa1def8]
> V [libjvm.so+0xa1f8d5]
> V [libjvm.so+0x7610f8] JVM_InvokeMethod+0x128
> J 2286
> sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (0 bytes) @ 0x7f7acb553ced [0x7f7acb553c00+0xed]
> J 6921 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7f7acbd1de38 [0x7f7acbd1ddc0+0x78]
> J 3645 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7f7acaf6e894 [0x7f7acaf6e640+0x254]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6af9ba]
> V [libjvm.so+0x72c046]
> V [libjvm.so+0x730523]
> C 0x7f7ab4c5d0d0
> C [impalad+0x26a2648]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x7a
> C [impalad+0x26a25cb]
> impala::ScalarExprEvaluator::GetValue(impala::TupleRow const*)+0x2b
> C [impalad+0x21f4f78]
> impala::AsciiQueryResultSet::AddRows(std::vector std::allocator > const&, impala::RowBatch*,
> int, int)+0x4c2
> C [impalad+0x25c5862]
> impala::BufferedPlanRootSink::GetNext(impala::RuntimeState*,
> impala::QueryResultSet*, int, bool*, long)+0x70c
> C [impalad+0x296cf17] impala::Coordinator::GetNext(impala::QueryResultSet*,
> int, bool*, long)+0x557
> C [impalad+0x219f5fe] impala::ClientRequestState::FetchRowsInternal(int,
> impala::QueryResultSet*, long)+0x6b2
> C [impalad+0x219d98e] impala::ClientRequestState::FetchRows(int,
> impala::QueryResultSet*, long)+0x46
> C [impalad+0x21c1d29]
> impala::ImpalaServer::FetchInternal(impala::TUniqueId, bool, int,
> beeswax::Results*)+0x717
> C [impalad+0x21bbde9] impala::ImpalaServer::fetch(beeswax::Results&,
> beeswax::QueryHandle const&, bool, int)+0x577
> {code}
> If result spooling is enabled, BufferedPlanRootSink will be used and
> ScalarExprEvaluation will be called in BufferedPlanRootSink::GetNext, leading
> to this crash.
> Without result spooling, BlockingPlanRootSink will be used and
> ScalarExprEvaluation is called in BlockingPlanRootSink::Send. No crash happen
> when result spooling is disabled.
> Attached is the full JVM crash log and resolved minidump.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10370) Thrift Message Incompatibility Detector
[ https://issues.apache.org/jira/browse/IMPALA-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241834#comment-17241834 ] Tim Armstrong commented on IMPALA-10370: [~jwjwyoung] thanks, this seems like a useful tool. Impala's use of Thrift is a bit unique in that it's used extensively for internal communication between C++/Java and within a cluster, where we don't support inter-version interoperability. So the thrift types that are only used internally aren't kept backwards compatible (attempting to do that somehow would require writing a bunch of unreachable code). It looks like we did break compatibility with TAggregateFunction/TFunction in the past and this caught that (that type is serialized and written to the hive metastore) but I don't think we can fix that at this point. > Thrift Message Incompatibility Detector > --- > > Key: IMPALA-10370 > URL: https://issues.apache.org/jira/browse/IMPALA-10370 > Project: IMPALA > Issue Type: Improvement >Reporter: junwen yang >Priority: Major > Attachments: impala_thrift_incompatibility.txt > > > Regarding the issue IMPALA-8243 caused by the incompatibility of thrift > message, we have created a static checker which keeps track of the thrift > file change, and detect potential incompatibility: > # Add/delete required field. The thrift guidelines suggests _Any new fields > that you add should be optional_. > # The tag number of a field has been changed. Also, the thrift guidelines > suggests _Don’t change the numeric tags for any existing fields_. > # A required field has been changed to optional, or an optional field has > been changed to required. According to the guidelines , _*Required Is > Forever* You should be very careful about marking fields as required. If at > some point you wish to stop writing or sending a required field, it will be > problematic to change the field to an optional field — old readers will > consider messages without this field to be incomplete and may reject or drop > them unintentionally. You should consider writing application-specific custom > validation routines for your buffers instead. Some have come to the > conclusion that using required does more harm than good; they prefer to use > only optional. However, this view is not universal._ > We have applied our checker on the frequently maintained IMPALA versions: > refs/tags/2.10.0, refs/tags/2.11.0, refs/tags/2.12.0, refs/tags/2.7.0, > refs/tags/2.8.0, refs/tags/2.9.0, refs/tags/3.0.0, refs/tags/3.0.1, > refs/tags/3.1.0, refs/tags/3.2.0, refs/tags/3.3.0, refs/tags/3.4.0, we found > more than 1000 problems as attached. > The results reported by our checker got confirmed by developers of HBASE and > our checker is requested by them, which can be found at HBASE-25340. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10366) TestRuntimeProfile.test_runtime_profile_aggregated failed on master core run
[
https://issues.apache.org/jira/browse/IMPALA-10366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10366.
Fix Version/s: Impala 4.0
Resolution: Fixed
> TestRuntimeProfile.test_runtime_profile_aggregated failed on master core run
>
>
> Key: IMPALA-10366
> URL: https://issues.apache.org/jira/browse/IMPALA-10366
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.4.0
>Reporter: Aman Sinha
>Assignee: Tim Armstrong
>Priority: Major
> Fix For: Impala 4.0
>
>
> Test failure on an impala-asf-master-core-erasure-coding build:
> {noformat}
> custom_cluster.test_runtime_profile.TestRuntimeProfile.test_runtime_profile_aggregated[protocol:
> beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none
> {noformat}
> The output is very verbose. Here's the first part of the Stacktrace:
> {noformat}
> custom_cluster/test_runtime_profile.py:46: in test_runtime_profile_aggregated
> self.run_test_case('runtime-profile-aggregated', vector)
> common/impala_test_suite.py:718: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:612: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*NumBackends: 4 .*
> E row_regex: .*Fragment F00 \[4 instances\].*
> E row_regex: .*Table Name\[0-3\]: tpch.*lineitem.*
> E
> E ACTUAL PROFILE:
> E Query (id=7b4476dc2da1ab29:ca76ce49):
> E DEBUG MODE WARNING: Query profile created while running a DEBUG build
> of Impala. Use RELEASE builds to measure query performance.
> E - InactiveTotalTime: 0.000ns
> E - TotalTime: 0.000ns
> E Summary:
> E Session ID: d94934f8d0ba24cb:79a237b3d0ef5a9a
> E Session Type: BEESWAX
> E Start Time: 2020-11-28 21:44:31.917844000
> E End Time: 2020-11-28 21:44:36.293303000
> E Query Type: QUERY
> E Query State: FINISHED
> E Impala Query State: FINISHED
> E Query Status: OK
> E Impala Version: impalad version 4.0.0-SNAPSHOT DEBUG (build
> a4cf449c88ef3fe08db9abbad82664b99382014c)
> E User: jenkins
> E Connected User: jenkins
> E Delegated User:
> E Network Address: ::1:45158
> E Default Db: tpch
> E Sql Statement: select STRAIGHT_JOIN count(distinct l_partkey)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10367) Impala-shell internal error - UnboundLocalError, local variable 'retry_msg' referenced before assign
Tim Armstrong created IMPALA-10367:
--
Summary: Impala-shell internal error - UnboundLocalError, local
variable 'retry_msg' referenced before assign
Key: IMPALA-10367
URL: https://issues.apache.org/jira/browse/IMPALA-10367
Project: IMPALA
Issue Type: Bug
Components: Clients
Affects Versions: Impala 4.0
Reporter: Tim Armstrong
Assignee: Abhishek Rawat
I saw this error when cancelling a query from impala-shell.
{noformat}
[localhost.EXAMPLE.COM:21050] tpch_parquet> explain select STRAIGHT_JOIN
count(distinct l_partkey)
from lineitem join [BROADCAST] supplier on l_suppkey = s_suppkey;
Query: explain select STRAIGHT_JOIN count(distinct l_partkey)
from lineitem join [BROADCAST] supplier on l_suppkey = s_suppkey
Caught exception [Errno 4] Interrupted system call, type=
in ExecuteStatement.
Cancelling Query
Caught exception [Errno 4] Interrupted system call, type=
in CloseSession.
Warning: close session RPC failed: [Errno 4] Interrupted system call,
Opened TCP connection to localhost.EXAMPLE.COM:21050
Caught exception [Errno 4] Interrupted system call, type=
in ExecuteStatement.
Caught exception [Errno 4] Interrupted system call, type=
in CloseSession.
Warning: close session RPC failed: [Errno 4] Interrupted system call,
Error connecting: UnboundLocalError, local variable 'retry_msg' referenced
before assign
{noformat}
This code was introduced by IMPALA-9466. I looked at the code briefly and
AFAICT it happens when the retry logic is disabled but we hit a
TTransportException or similar error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-10366) TestRuntimeProfile.test_runtime_profile_aggregated failed on master core run
[
https://issues.apache.org/jira/browse/IMPALA-10366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-10366 started by Tim Armstrong.
--
> TestRuntimeProfile.test_runtime_profile_aggregated failed on master core run
>
>
> Key: IMPALA-10366
> URL: https://issues.apache.org/jira/browse/IMPALA-10366
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.4.0
>Reporter: Aman Sinha
>Assignee: Tim Armstrong
>Priority: Major
>
> Test failure on an impala-asf-master-core-erasure-coding build:
> {noformat}
> custom_cluster.test_runtime_profile.TestRuntimeProfile.test_runtime_profile_aggregated[protocol:
> beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none
> {noformat}
> The output is very verbose. Here's the first part of the Stacktrace:
> {noformat}
> custom_cluster/test_runtime_profile.py:46: in test_runtime_profile_aggregated
> self.run_test_case('runtime-profile-aggregated', vector)
> common/impala_test_suite.py:718: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:612: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*NumBackends: 4 .*
> E row_regex: .*Fragment F00 \[4 instances\].*
> E row_regex: .*Table Name\[0-3\]: tpch.*lineitem.*
> E
> E ACTUAL PROFILE:
> E Query (id=7b4476dc2da1ab29:ca76ce49):
> E DEBUG MODE WARNING: Query profile created while running a DEBUG build
> of Impala. Use RELEASE builds to measure query performance.
> E - InactiveTotalTime: 0.000ns
> E - TotalTime: 0.000ns
> E Summary:
> E Session ID: d94934f8d0ba24cb:79a237b3d0ef5a9a
> E Session Type: BEESWAX
> E Start Time: 2020-11-28 21:44:31.917844000
> E End Time: 2020-11-28 21:44:36.293303000
> E Query Type: QUERY
> E Query State: FINISHED
> E Impala Query State: FINISHED
> E Query Status: OK
> E Impala Version: impalad version 4.0.0-SNAPSHOT DEBUG (build
> a4cf449c88ef3fe08db9abbad82664b99382014c)
> E User: jenkins
> E Connected User: jenkins
> E Delegated User:
> E Network Address: ::1:45158
> E Default Db: tpch
> E Sql Statement: select STRAIGHT_JOIN count(distinct l_partkey)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8962) FETCH_ROWS_TIMEOUT_MS should apply before rows are available
[
https://issues.apache.org/jira/browse/IMPALA-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241021#comment-17241021
]
Tim Armstrong commented on IMPALA-8962:
---
[~arawat] [~asherman] is this something we've seen before with JDBC?
> FETCH_ROWS_TIMEOUT_MS should apply before rows are available
>
>
> Key: IMPALA-8962
> URL: https://issues.apache.org/jira/browse/IMPALA-8962
> Project: IMPALA
> Issue Type: Bug
> Components: Clients
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: Impala 3.4.0
>
>
> IMPALA-7312 added a fetch timeout controlled by the query option
> {{FETCH_ROWS_TIMEOUT_MS}}. The issue is that the timeout only applies after
> the *first* batch of rows are available. The issue is that both Beeswax and
> HS2 clients call {{request_state->BlockOnWait}} inside
> {{ImpalaServer::FetchInternal}}. The call to {{BlockOnWait}} blocks until
> rows are ready to be consumed via {{ClientRequestState::FetchRows}}.
> So clients can still end up blocking indefinitely waiting for the first row
> batch to appear.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-1792) ImpalaODBC: Can not get the value in the SQLGetData(m-x th column) after the SQLBindCol(m th column)
[
https://issues.apache.org/jira/browse/IMPALA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241011#comment-17241011
]
Tim Armstrong commented on IMPALA-1792:
---
[^odbc.c] [^odbc.ini] [^odbcinst.ini]
{noformat}
gcc odbc.c -L/opt/cloudera/impalaodbc/lib/64/ -lclouderaimpalaodbc64 -o odbc
ODBCINI=~/odbc.ini ODBCINSTINI=~/odbcinst.ini
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libodbcinst.so \
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/cloudera/impalaodbc/lib/64/ \
./odbc
{noformat}
> ImpalaODBC: Can not get the value in the SQLGetData(m-x th column) after the
> SQLBindCol(m th column)
>
>
> Key: IMPALA-1792
> URL: https://issues.apache.org/jira/browse/IMPALA-1792
> Project: IMPALA
> Issue Type: Bug
> Components: Clients
>Affects Versions: Impala 2.1
> Environment: OS: CentOS release 6.4 (Final)
> Impala: 2.1.0
> ImpalaODBC: 2.5.23
>Reporter: Mitsuhiro Koga
>Priority: Critical
> Labels: correctness, downgraded, impala, odbc
> Attachments: odbc.c, odbc.ini, odbcinst.ini
>
>
> Steps to reproduce
> # Create table and insert data in impala.
> {code:sql}
> create table t (c1 string, c2 string, c3 string);
> insert into t (c1, c2, c3) values ('AAA', 'BBB', 'CCC');
> {code}
> # Query the t table with the following code.
> {code}
> #include
> #include
> #include
> #include
> int main() {
> SQLHENV env;
> SQLHDBC dbc;
> SQLHSTMT stmt;
> SQLRETURN ret;
> SQLAllocHandle(SQL_HANDLE_ENV, SQL_NULL_HANDLE, );
> SQLSetEnvAttr(env, SQL_ATTR_ODBC_VERSION, (void *) SQL_OV_ODBC3, 0);
> SQLAllocHandle(SQL_HANDLE_DBC, env, );
> SQLDriverConnect(dbc, (SQLCHAR*)NULL, (SQLCHAR*)"DSN=Impala", SQL_NTS,
> NULL, 0, NULL, SQL_DRIVER_COMPLETE);
> SQLAllocHandle(SQL_HANDLE_STMT, dbc, );
> SQLExecDirect(stmt, (SQLCHAR*)"select c1, c2, c3 from t", SQL_NTS);
> char szCol1[11];
> char szCol2[11];
> char szCol3[11];
> SQLLEN nColLen1;
> SQLLEN nColLen2;
> SQLLEN nColLen3;
> /*/
> / Bind 2nd column /
> /*/
> SQLBindCol(stmt, 2, SQL_C_CHAR, szCol2, sizeof(szCol2), );
> ret = SQLFetch(stmt);
> if (ret == SQL_SUCCESS || ret == SQL_SUCCESS_WITH_INFO) {
> //
> / Get 1st column /
> //
> ret = SQLGetData(stmt, 1, SQL_C_CHAR, szCol1, sizeof(szCol1),
> );
> if (ret == SQL_SUCCESS || ret == SQL_SUCCESS_WITH_INFO) {
> printf("c1: %s\n", szCol1);
> } else {
> printf("no data\n");
> }
> printf("c2: %s\n", szCol2);
> //
> / Get 3rd column /
> //
> ret = SQLGetData(stmt, 3, SQL_C_CHAR, szCol3, sizeof(szCol3),
> );
> if (ret == SQL_SUCCESS || ret == SQL_SUCCESS_WITH_INFO) {
> printf("c3: %s\n", szCol3);
> } else {
> printf("no data\n");
> }
> } else {
> printf("no row\n");
> }
> return 0;
> }
> {code}
> Expected result
> {code}
> c1: AAA
> c2: BBB
> c3: CCC
> {code}
> Actual result
> {code}
> no data
> c2: BBB
> c3: CCC
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-1792) ImpalaODBC: Can not get the value in the SQLGetData(m-x th column) after the SQLBindCol(m th column)
[
https://issues.apache.org/jira/browse/IMPALA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241007#comment-17241007
]
Tim Armstrong commented on IMPALA-1792:
---
I tested this with ODBC driver version 2.6.11 and got the expected output, so
it is fixed in the driver.
> ImpalaODBC: Can not get the value in the SQLGetData(m-x th column) after the
> SQLBindCol(m th column)
>
>
> Key: IMPALA-1792
> URL: https://issues.apache.org/jira/browse/IMPALA-1792
> Project: IMPALA
> Issue Type: Bug
> Components: Clients
>Affects Versions: Impala 2.1
> Environment: OS: CentOS release 6.4 (Final)
> Impala: 2.1.0
> ImpalaODBC: 2.5.23
>Reporter: Mitsuhiro Koga
>Priority: Critical
> Labels: correctness, downgraded, impala, odbc
> Attachments: odbc.c, odbc.ini, odbcinst.ini
>
>
> Steps to reproduce
> # Create table and insert data in impala.
> {code:sql}
> create table t (c1 string, c2 string, c3 string);
> insert into t (c1, c2, c3) values ('AAA', 'BBB', 'CCC');
> {code}
> # Query the t table with the following code.
> {code}
> #include
> #include
> #include
> #include
> int main() {
> SQLHENV env;
> SQLHDBC dbc;
> SQLHSTMT stmt;
> SQLRETURN ret;
> SQLAllocHandle(SQL_HANDLE_ENV, SQL_NULL_HANDLE, );
> SQLSetEnvAttr(env, SQL_ATTR_ODBC_VERSION, (void *) SQL_OV_ODBC3, 0);
> SQLAllocHandle(SQL_HANDLE_DBC, env, );
> SQLDriverConnect(dbc, (SQLCHAR*)NULL, (SQLCHAR*)"DSN=Impala", SQL_NTS,
> NULL, 0, NULL, SQL_DRIVER_COMPLETE);
> SQLAllocHandle(SQL_HANDLE_STMT, dbc, );
> SQLExecDirect(stmt, (SQLCHAR*)"select c1, c2, c3 from t", SQL_NTS);
> char szCol1[11];
> char szCol2[11];
> char szCol3[11];
> SQLLEN nColLen1;
> SQLLEN nColLen2;
> SQLLEN nColLen3;
> /*/
> / Bind 2nd column /
> /*/
> SQLBindCol(stmt, 2, SQL_C_CHAR, szCol2, sizeof(szCol2), );
> ret = SQLFetch(stmt);
> if (ret == SQL_SUCCESS || ret == SQL_SUCCESS_WITH_INFO) {
> //
> / Get 1st column /
> //
> ret = SQLGetData(stmt, 1, SQL_C_CHAR, szCol1, sizeof(szCol1),
> );
> if (ret == SQL_SUCCESS || ret == SQL_SUCCESS_WITH_INFO) {
> printf("c1: %s\n", szCol1);
> } else {
> printf("no data\n");
> }
> printf("c2: %s\n", szCol2);
> //
> / Get 3rd column /
> //
> ret = SQLGetData(stmt, 3, SQL_C_CHAR, szCol3, sizeof(szCol3),
> );
> if (ret == SQL_SUCCESS || ret == SQL_SUCCESS_WITH_INFO) {
> printf("c3: %s\n", szCol3);
> } else {
> printf("no data\n");
> }
> } else {
> printf("no row\n");
> }
> return 0;
> }
> {code}
> Expected result
> {code}
> c1: AAA
> c2: BBB
> c3: CCC
> {code}
> Actual result
> {code}
> no data
> c2: BBB
> c3: CCC
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-1792) ImpalaODBC: Can not get the value in the SQLGetData(m-x th column) after the SQLBindCol(m th column)
[
https://issues.apache.org/jira/browse/IMPALA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-1792:
--
Attachment: odbcinst.ini
odbc.ini
odbc.c
> ImpalaODBC: Can not get the value in the SQLGetData(m-x th column) after the
> SQLBindCol(m th column)
>
>
> Key: IMPALA-1792
> URL: https://issues.apache.org/jira/browse/IMPALA-1792
> Project: IMPALA
> Issue Type: Bug
> Components: Clients
>Affects Versions: Impala 2.1
> Environment: OS: CentOS release 6.4 (Final)
> Impala: 2.1.0
> ImpalaODBC: 2.5.23
>Reporter: Mitsuhiro Koga
>Priority: Critical
> Labels: correctness, downgraded, impala, odbc
> Attachments: odbc.c, odbc.ini, odbcinst.ini
>
>
> Steps to reproduce
> # Create table and insert data in impala.
> {code:sql}
> create table t (c1 string, c2 string, c3 string);
> insert into t (c1, c2, c3) values ('AAA', 'BBB', 'CCC');
> {code}
> # Query the t table with the following code.
> {code}
> #include
> #include
> #include
> #include
> int main() {
> SQLHENV env;
> SQLHDBC dbc;
> SQLHSTMT stmt;
> SQLRETURN ret;
> SQLAllocHandle(SQL_HANDLE_ENV, SQL_NULL_HANDLE, );
> SQLSetEnvAttr(env, SQL_ATTR_ODBC_VERSION, (void *) SQL_OV_ODBC3, 0);
> SQLAllocHandle(SQL_HANDLE_DBC, env, );
> SQLDriverConnect(dbc, (SQLCHAR*)NULL, (SQLCHAR*)"DSN=Impala", SQL_NTS,
> NULL, 0, NULL, SQL_DRIVER_COMPLETE);
> SQLAllocHandle(SQL_HANDLE_STMT, dbc, );
> SQLExecDirect(stmt, (SQLCHAR*)"select c1, c2, c3 from t", SQL_NTS);
> char szCol1[11];
> char szCol2[11];
> char szCol3[11];
> SQLLEN nColLen1;
> SQLLEN nColLen2;
> SQLLEN nColLen3;
> /*/
> / Bind 2nd column /
> /*/
> SQLBindCol(stmt, 2, SQL_C_CHAR, szCol2, sizeof(szCol2), );
> ret = SQLFetch(stmt);
> if (ret == SQL_SUCCESS || ret == SQL_SUCCESS_WITH_INFO) {
> //
> / Get 1st column /
> //
> ret = SQLGetData(stmt, 1, SQL_C_CHAR, szCol1, sizeof(szCol1),
> );
> if (ret == SQL_SUCCESS || ret == SQL_SUCCESS_WITH_INFO) {
> printf("c1: %s\n", szCol1);
> } else {
> printf("no data\n");
> }
> printf("c2: %s\n", szCol2);
> //
> / Get 3rd column /
> //
> ret = SQLGetData(stmt, 3, SQL_C_CHAR, szCol3, sizeof(szCol3),
> );
> if (ret == SQL_SUCCESS || ret == SQL_SUCCESS_WITH_INFO) {
> printf("c3: %s\n", szCol3);
> } else {
> printf("no data\n");
> }
> } else {
> printf("no row\n");
> }
> return 0;
> }
> {code}
> Expected result
> {code}
> c1: AAA
> c2: BBB
> c3: CCC
> {code}
> Actual result
> {code}
> no data
> c2: BBB
> c3: CCC
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10351) Enable mt_dop for DML
[ https://issues.apache.org/jira/browse/IMPALA-10351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-10351. Fix Version/s: Impala 4.0 Resolution: Fixed > Enable mt_dop for DML > - > > Key: IMPALA-10351 > URL: https://issues.apache.org/jira/browse/IMPALA-10351 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: multithreading > Fix For: Impala 4.0 > > > We don't have any pending issues for MT_DOP with DML and have sufficient > functional testing, so I think we can go ahead and enable it. IMPALA-8125 and > query hints give us a safety valve in case we overparallelize. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9812) Remove --unlock_mt_dop and--mt_dop_auto_fallback
[ https://issues.apache.org/jira/browse/IMPALA-9812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-9812. --- Fix Version/s: Impala 4.0 Resolution: Fixed > Remove --unlock_mt_dop and--mt_dop_auto_fallback > - > > Key: IMPALA-9812 > URL: https://issues.apache.org/jira/browse/IMPALA-9812 > Project: IMPALA > Issue Type: Sub-task >Reporter: Tim Armstrong >Priority: Minor > Fix For: Impala 4.0 > > > These flags will become ineffective when DML is supported. We should clean up > all references and move them to the flag graveyard. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10216) BufferPoolTest.WriteErrorBlacklistCompression is flaky on UBSAN builds
[
https://issues.apache.org/jira/browse/IMPALA-10216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10216.
Fix Version/s: Impala 4.0
Resolution: Fixed
> BufferPoolTest.WriteErrorBlacklistCompression is flaky on UBSAN builds
> --
>
> Key: IMPALA-10216
> URL: https://issues.apache.org/jira/browse/IMPALA-10216
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: flaky
> Fix For: Impala 4.0
>
> Attachments: LastTest.log, LastTestsFailed.log,
> buffer-pool-test.ERROR, buffer-pool-test.INFO, buffer-pool-test.WARNING,
> buffer-pool-test.xml, impala-cdpd-master-staging-core-tsan_111.tar.gz
>
>
> Only seen this once so far:
> {code}
> BufferPoolTest.WriteErrorBlacklistCompression
> Error Message
> Value of: FindPageInDir(pages[NO_ERROR_QUERY], error_dir) != NULL
> Actual: false
> Expected: true
> Stacktrace
> Impala/be/src/runtime/bufferpool/buffer-pool-test.cc:1764
> Value of: FindPageInDir(pages[NO_ERROR_QUERY], error_dir) != NULL
> Actual: false
> Expected: true
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10357) A test case in test_basic_joins() seems flaky
[
https://issues.apache.org/jira/browse/IMPALA-10357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10357.
Resolution: Duplicate
This looks like a duplicate of IMPALA-10336 - it's not returning the expected
error but instead this incorrect error.
> A test case in test_basic_joins() seems flaky
> -
>
> Key: IMPALA-10357
> URL: https://issues.apache.org/jira/browse/IMPALA-10357
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Fang-Yu Rao
>Assignee: Tim Armstrong
>Priority: Major
> Labels: broken-build, flaky-test
> Attachments: excerpted_impalad.INFO, excerpted_impalad_node1.INFO
>
>
> We found in a recent build that a test case in
> [test_basic_joins()|https://github.com/apache/impala/blob/master/tests/query_test/test_join_queries.py#L71]
> does not fail with the expected error. This test reads the test case at
> [joins.test|https://github.com/apache/impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/joins.test#L846-L850]
> and verifies if the returned result is "{{Debug Action: FAIL}}".
> {noformat}
> Error Message
> query_test/test_join_queries.py:71: in test_basic_joins
> self.run_test_case('QueryTest/joins', new_vector)
> common/impala_test_suite.py:668: in run_test_case
> self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> common/impala_test_suite.py:485: in __verify_exceptions (expected_str,
> actual_str) E AssertionError: Unexpected exception string. Expected: Debug
> Action: FAIL E Not found in actual: ImpalaBeeswaxException: Query
> aborted:ExecQueryFInstances rpc query_id=9f4e869839247e75:d01b0093
> failed: Exec() rpc failed: Aborted: ExecQueryFInstances RPC to
> 127.0.0.1:27002 is cancelled in state SENT
> {noformat}
> {noformat}
> Stacktrace
> query_test/test_join_queries.py:71: in test_basic_joins
> self.run_test_case('QueryTest/joins', new_vector)
> common/impala_test_suite.py:668: in run_test_case
> self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> common/impala_test_suite.py:485: in __verify_exceptions
> (expected_str, actual_str)
> E AssertionError: Unexpected exception string. Expected: Debug Action: FAIL
> E Not found in actual: ImpalaBeeswaxException: Query
> aborted:ExecQueryFInstances rpc query_id=9f4e869839247e75:d01b0093
> failed: Exec() rpc failed: Aborted: ExecQueryFInstances RPC to
> 127.0.0.1:27002 is cancelled in state SENT
> {noformat}
> {noformat}
> Standard Error
> set debug_action="2:PREPARE:FAIL|COORD_BEFORE_EXEC_RPC:JITTER@100@0.3";
> -- 2020-11-23 23:27:30,175 INFO MainThread: Started query
> 1a49038843b6b4da:30f62281
> -- executing against localhost:21000
> select 1 from alltypestiny a join alltypestiny b on a.id != b.id;
> -- 2020-11-23 23:27:30,188 INFO MainThread: Started query
> 9f4e869839247e75:d01b0093
> -- executing against localhost:21000
> {noformat}
> According to the output to standard error, two related queries are of query
> id's "{{1a49038843b6b4da:30f62281}}" and
> "{{9f4e869839247e75:d01b0093}}".
> Using these two query id's, we excerpted the related log entries from two log
> files of impalad's.
> Since this test was recently revised in IMPALA-9309, maybe [~tarmstrong]
> could offer some insight into it. Assigned it to Tim for now but feel free to
> re-assign it as you see appropriate. Also let me know if some other logs are
> needed for investigation. Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-10216) BufferPoolTest.WriteErrorBlacklistCompression is flaky on UBSAN builds
[
https://issues.apache.org/jira/browse/IMPALA-10216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-10216 started by Tim Armstrong.
--
> BufferPoolTest.WriteErrorBlacklistCompression is flaky on UBSAN builds
> --
>
> Key: IMPALA-10216
> URL: https://issues.apache.org/jira/browse/IMPALA-10216
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: flaky
> Attachments: LastTest.log, LastTestsFailed.log,
> buffer-pool-test.ERROR, buffer-pool-test.INFO, buffer-pool-test.WARNING,
> buffer-pool-test.xml, impala-cdpd-master-staging-core-tsan_111.tar.gz
>
>
> Only seen this once so far:
> {code}
> BufferPoolTest.WriteErrorBlacklistCompression
> Error Message
> Value of: FindPageInDir(pages[NO_ERROR_QUERY], error_dir) != NULL
> Actual: false
> Expected: true
> Stacktrace
> Impala/be/src/runtime/bufferpool/buffer-pool-test.cc:1764
> Value of: FindPageInDir(pages[NO_ERROR_QUERY], error_dir) != NULL
> Actual: false
> Expected: true
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10216) BufferPoolTest.WriteErrorBlacklistCompression is flaky on UBSAN builds
[
https://issues.apache.org/jira/browse/IMPALA-10216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238358#comment-17238358
]
Tim Armstrong commented on IMPALA-10216:
So from the failed assertion we have:
{noformat}
TmpFilePaths(pages[NO_ERROR_QUERY])
/tmp/buffer-pool-test.1/impala-scratch/:_3083829c-75fc-4c44-812a-bb5a4c067c88
/tmp/buffer-pool-test.1/impala-scratch/:_3083829c-75fc-4c44-812a-bb5a4c067c88
DumpScratchDir(error_dir)
/tmp/buffer-pool-test.0/impala-scratch:
:_b049d991-adb2-4845-991f-2dc287c0c90a
:_20081b32-bef6-4909-8bff-4fd61ec8ca0d
{noformat}
buffer-pool-test.INFO for reference (it looks normal)
{noformat}
I1123 23:09:20.500730 9357 tmp-file-mgr.cc:254] Using scratch directory
/tmp/buffer-pool-test.0/impala-scratch on disk 0 limit: 8589934592.00 GB
I1123 23:09:20.506762 9357 tmp-file-mgr.cc:254] Using scratch directory
/tmp/buffer-pool-test.1/impala-scratch on disk 0 limit: 8589934592.00 GB
I1123 23:09:20.545521 9357 buffer-pool-test.cc:1727] Manager 0 Block 0 backed
by file
/tmp/buffer-pool-test.0/impala-scratch/:_b049d991-adb2-4845-991f-2dc287c0c90a
I1123 23:09:20.545562 9357 buffer-pool-test.cc:1727] Manager 0 Block 1 backed
by file
/tmp/buffer-pool-test.1/impala-scratch/:_cd33af91-4e27-412a-ae45-f16061b5ef56
I1123 23:09:20.556377 9357 buffer-pool-test.cc:1727] Manager 1 Block 0 backed
by file
/tmp/buffer-pool-test.1/impala-scratch/:_3083829c-75fc-4c44-812a-bb5a4c067c88
I1123 23:09:20.556414 9357 buffer-pool-test.cc:1727] Manager 1 Block 1 backed
by file
/tmp/buffer-pool-test.0/impala-scratch/:_20081b32-bef6-4909-8bff-4fd61ec8ca0d
I1123 23:09:20.558903 9357 buffer-pool-test.cc:336] Injected fault by removing
file permissions
/tmp/buffer-pool-test.0/impala-scratch/:_b049d991-adb2-4845-991f-2dc287c0c90a
E1123 23:09:20.559283 451 tmp-file-mgr.cc:359] Error for temporary file
'/tmp/buffer-pool-test.0/impala-scratch/:_b049d991-adb2-4845-991f-2dc287c0c90a':
Disk I/O error on
impala-ec2-centos74-m5-4xlarge-ondemand-1898.vpc.cloudera.com:27000: open()
failed for
/tmp/buffer-pool-test.0/impala-scratch/:_b049d991-adb2-4845-991f-2dc287c0c90a.
Access denied for the process' user errno=13
I1123 23:09:20.563634 9357 buffer-pool-test.cc:1781] Newly created page backed
by file
/tmp/buffer-pool-test.1/impala-scratch/:_cd33af91-4e27-412a-ae45-f16061b5ef56
I1123 23:09:20.563666 9357 buffer-pool-test.cc:1781] Newly created page backed
by file
/tmp/buffer-pool-test.1/impala-scratch/:_cd33af91-4e27-412a-ae45-f16061b5ef56
I1123 23:09:20.606637 9357 krpc-data-stream-mgr.cc:436] Waiting for
data-stream-mgr maintenance thread...
I1123 23:09:20.606671 9357 krpc-data-stream-mgr.cc:438] Waiting for
deserialization thread pool...
{noformat}
It's weird that both pages got backed by the same temp dir, because the
allocation should be round-robin. II looked at the code and I do think there's
some actual non-determinism here because of the way Pin() runs asynchronously
until the actual buffer is requested - it may not actually complete for one of
these pages. I think if one thread went down this code path on one page but not
the other, the allocation may not be round robin -
https://github.com/apache/impala/blob/master/be/src/runtime/bufferpool/buffer-pool.cc#L207
> BufferPoolTest.WriteErrorBlacklistCompression is flaky on UBSAN builds
> --
>
> Key: IMPALA-10216
> URL: https://issues.apache.org/jira/browse/IMPALA-10216
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: flaky
> Attachments: LastTest.log, LastTestsFailed.log,
> buffer-pool-test.ERROR, buffer-pool-test.INFO, buffer-pool-test.WARNING,
> buffer-pool-test.xml, impala-cdpd-master-staging-core-tsan_111.tar.gz
>
>
> Only seen this once so far:
> {code}
> BufferPoolTest.WriteErrorBlacklistCompression
> Error Message
> Value of: FindPageInDir(pages[NO_ERROR_QUERY], error_dir) != NULL
> Actual: false
> Expected: true
> Stacktrace
> Impala/be/src/runtime/bufferpool/buffer-pool-test.cc:1764
> Value of: FindPageInDir(pages[NO_ERROR_QUERY], error_dir) != NULL
> Actual: false
> Expected: true
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
[jira] [Commented] (IMPALA-10216) BufferPoolTest.WriteErrorBlacklistCompression is flaky on UBSAN builds
[
https://issues.apache.org/jira/browse/IMPALA-10216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238350#comment-17238350
]
Tim Armstrong commented on IMPALA-10216:
[~fangyurao] thanks for attaching, I'll take a look.
FWIW the error logs in the ERROR file are likely unrelated - these tests do
deliberately induce errors so that's expected.
> BufferPoolTest.WriteErrorBlacklistCompression is flaky on UBSAN builds
> --
>
> Key: IMPALA-10216
> URL: https://issues.apache.org/jira/browse/IMPALA-10216
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: flaky
> Attachments: LastTest.log, LastTestsFailed.log,
> buffer-pool-test.ERROR, buffer-pool-test.INFO, buffer-pool-test.WARNING,
> buffer-pool-test.xml, impala-cdpd-master-staging-core-tsan_111.tar.gz
>
>
> Only seen this once so far:
> {code}
> BufferPoolTest.WriteErrorBlacklistCompression
> Error Message
> Value of: FindPageInDir(pages[NO_ERROR_QUERY], error_dir) != NULL
> Actual: false
> Expected: true
> Stacktrace
> Impala/be/src/runtime/bufferpool/buffer-pool-test.cc:1764
> Value of: FindPageInDir(pages[NO_ERROR_QUERY], error_dir) != NULL
> Actual: false
> Expected: true
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10356) Analyzed query in explain plan is not quite right for insert with values clause
Tim Armstrong created IMPALA-10356:
--
Summary: Analyzed query in explain plan is not quite right for
insert with values clause
Key: IMPALA-10356
URL: https://issues.apache.org/jira/browse/IMPALA-10356
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 4.0
Reporter: Tim Armstrong
In impala-shell:
{noformat}
create table double_tbl (d double) stored as textfile;
set explain_level=2;
explain insert into double_tbl values (-0.43149576573887316);
{noformat}
{noformat}
+--+
| Explain String
|
+--+
| Max Per-Host Resource Reservation: Memory=0B Threads=1
|
| Per-Host Resource Estimates: Memory=10MB
|
| Codegen disabled by planner
|
| Analyzed query: SELECT CAST(-0.43149576573887316 AS DECIMAL(17,17)) UNION
SELECT |
| CAST(-0.43149576573887316 AS DECIMAL(17,17))
|
|
|
| F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
|
| | Per-Host Resources: mem-estimate=8B mem-reservation=0B
thread-reservation=1 |
| WRITE TO HDFS [default.double_tbl, OVERWRITE=false]
|
| | partitions=1
|
| | output exprs: CAST(-0.43149576573887316 AS DOUBLE)
|
| | mem-estimate=8B mem-reservation=0B thread-reservation=0
|
| |
|
| 00:UNION
|
|constant-operands=1
|
|mem-estimate=0B mem-reservation=0B thread-reservation=0
|
|tuple-ids=0 row-size=8B cardinality=1
|
|in pipelines:
|
+--+
{noformat}
The analyzed query does not make sense. We should investigate and fix it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9453) S3 build failed with many strange symptoms
[
https://issues.apache.org/jira/browse/IMPALA-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-9453:
--
Priority: Critical (was: Blocker)
> S3 build failed with many strange symptoms
> --
>
> Key: IMPALA-9453
> URL: https://issues.apache.org/jira/browse/IMPALA-9453
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.4.0
>Reporter: Tim Armstrong
>Priority: Critical
> Labels: broken-build, crash
> Attachments: impalad.ERROR, impalad_node1.ERROR, impalad_node2.ERROR
>
>
> There were a lot of incorrect results:
> {noformat}
> uery_test/test_mt_dop.py:49: in test_mt_dop
> self.run_test_case('QueryTest/mt-dop', new_vector)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder) common/test_result_verifier.py:456: in
> verify_raw_results VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results E assert Comparing
> QueryTestResults (expected vs actual): E 7300 != 6990
> Stacktrace
> query_test/test_mt_dop.py:49: in test_mt_dop
> self.run_test_case('QueryTest/mt-dop', new_vector)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 7300 != 6990
> Standard Error
> ERROR:test_configuration:Comparing QueryTestResults (expected vs actual):
> 7300 != 6990
> {noformat}
> The impalads eventually crashed:
> {noformat}
> F0302 00:50:55.607841 483 parquet-page-reader.cc:67]
> e24eb0839fa75423:8ac0bf730002] Check failed: col_end <
> file_desc.file_length (7010 vs. 7010)
> *** Check failure stack trace: ***
> @ 0x4f7277c google::LogMessage::Fail()
> @ 0x4f74021 google::LogMessage::SendToLog()
> @ 0x4f72156 google::LogMessage::Flush()
> @ 0x4f7571d google::LogMessageFatal::~LogMessageFatal()
> @ 0x2e3a520 impala::ParquetPageReader::InitColumnChunk()
> @ 0x2e37dee impala::ParquetColumnChunkReader::InitColumnChunk()
> @ 0x2cd8000 impala::BaseScalarColumnReader::Reset()
> @ 0x2c91239 impala::HdfsParquetScanner::InitScalarColumns()
> @ 0x2c8775a impala::HdfsParquetScanner::NextRowGroup()
> @ 0x2c85c2a impala::HdfsParquetScanner::GetNextInternal()
> @ 0x2c8403e impala::HdfsParquetScanner::ProcessSplit()
> @ 0x28b826b impala::HdfsScanNode::ProcessSplit()
> @ 0x28b7440 impala::HdfsScanNode::ScannerThread()
> @ 0x28b679d
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x28b8d91
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x20aa1e9 boost::function0<>::operator()()
> @ 0x266a84a impala::Thread::SuperviseThread()
> @ 0x2672ace boost::_bi::list5<>::operator()<>()
> @ 0x26729f2 boost::_bi::bind_t<>::operator()()
> @ 0x26729b5 boost::detail::thread_data<>::run()
> @ 0x3e98b19 thread_proxy
> @ 0x7f8c2ccefe24 start_thread
> @ 0x7f8c2985b34c __clone
> {noformat}
> {noformat}
> F0302 00:50:41.466794 32643 parquet-page-reader.cc:67]
> dd48a46583bea9c8:e3be6412] Check failed: col_end <
> file_desc.file_length (7010 vs. 7010)
> *** Check failure stack trace: ***
> @ 0x4f7277c google::LogMessage::Fail()
> @ 0x4f74021 google::LogMessage::SendToLog()
> @ 0x4f72156 google::LogMessage::Flush()
> @ 0x4f7571d google::LogMessageFatal::~LogMessageFatal()
> @ 0x2e3a520 impala::ParquetPageReader::InitColumnChunk()
> @ 0x2e37dee impala::ParquetColumnChunkReader::InitColumnChunk()
> @ 0x2cd8000 impala::BaseScalarColumnReader::Reset()
> @ 0x2c91239 impala::HdfsParquetScanner::InitScalarColumns()
> @ 0x2c8775a impala::HdfsParquetScanner::NextRowGroup()
> @ 0x2c85c2a
[jira] [Resolved] (IMPALA-9050) test_scanners.TestScanRangeLengths.test_scan_ranges is flaky for kudu
[
https://issues.apache.org/jira/browse/IMPALA-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-9050.
---
Fix Version/s: Impala 4.0
Resolution: Fixed
> test_scanners.TestScanRangeLengths.test_scan_ranges is flaky for kudu
> -
>
> Key: IMPALA-9050
> URL: https://issues.apache.org/jira/browse/IMPALA-9050
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.4.0
>Reporter: Yongzhi Chen
>Assignee: Tim Armstrong
>Priority: Major
> Labels: build-failure, flaky
> Fix For: Impala 4.0
>
>
> Pre-commit build failed for unrelated change:
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/1405/testReport/junit/query_test.test_scanners/TestScanRangeLengths/test_scan_ranges_max_scan_range_length__512___protocol__beeswax___exec_optionbatch_size___0___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0table_format__kudu_none_/
> query_test.test_scanners.TestScanRangeLengths.test_scan_ranges[max_scan_range_length:
> 512 | protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> kudu/none]
> Error Message
> query_test/test_scanners.py:937: in test_scan_ranges
> self.run_test_case('QueryTest/hdfs-tiny-scan', vector)
> common/impala_test_suite.py:621: in run_test_case result = exec_fn(query,
> user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:556: in __exec_in_impala result =
> self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:893: in __execute_query return
> impalad_client.execute(query, user=user) common/impala_connection.py:205: in
> execute return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:205: in execute result =
> self.fetch_results(query_string, handle) beeswax/impala_beeswax.py:451: in
> fetch_results exec_result = self.__fetch_results(query_handle, max_rows)
> beeswax/impala_beeswax.py:462: in __fetch_results results =
> self.__do_rpc(lambda: self.imp_service.fetch(handle, False, fetch_rows))
> beeswax/impala_beeswax.py:519: in __do_rpc raise
> ImpalaBeeswaxException(self.__build_error_message(b), b) E
> ImpalaBeeswaxException: ImpalaBeeswaxException: EINNER EXCEPTION: 'beeswaxd.ttypes.BeeswaxException'> EMESSAGE: Unable to open Kudu table:
> Network error: failed to read from TLS socket (remote: 172.18.0.1:7051):
> Cannot send after transport endpoint shutdown (error 108)
> Stacktrace
> query_test/test_scanners.py:937: in test_scan_ranges
> self.run_test_case('QueryTest/hdfs-tiny-scan', vector)
> common/impala_test_suite.py:621: in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:556: in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:893: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:205: in execute
> result = self.fetch_results(query_string, handle)
> beeswax/impala_beeswax.py:451: in fetch_results
> exec_result = self.__fetch_results(query_handle, max_rows)
> beeswax/impala_beeswax.py:462: in __fetch_results
> results = self.__do_rpc(lambda: self.imp_service.fetch(handle, False,
> fetch_rows))
> beeswax/impala_beeswax.py:519: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: Unable to open Kudu table: Network error: failed to read from
> TLS socket (remote: 172.18.0.1:7051): Cannot send after transport endpoint
> shutdown (error 108)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-4238) custom_cluster/test_client_ssl.py TestClientSsl.test_ssl AssertionError: SIGINT was not caught by shell within 30s
[
https://issues.apache.org/jira/browse/IMPALA-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-4238.
---
Fix Version/s: Impala 4.0
Resolution: Fixed
> custom_cluster/test_client_ssl.py TestClientSsl.test_ssl AssertionError:
> SIGINT was not caught by shell within 30s
> --
>
> Key: IMPALA-4238
> URL: https://issues.apache.org/jira/browse/IMPALA-4238
> Project: IMPALA
> Issue Type: Bug
> Components: Security
>Affects Versions: Impala 2.8.0, Impala 2.10.0
>Reporter: Harrison Sheinblatt
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
> Fix For: Impala 4.0
>
>
> asf master core test failure:
> http://sandbox.jenkins.sf.cloudera.com/view/Impala/view/Evergreen-asf-master/job/impala-asf-master-core/540/
> http://sandbox.jenkins.sf.cloudera.com/job/impala-umbrella-build-and-test/4921/console
> {noformat}
> 08:18:51 === FAILURES
> ===
> 08:18:51 TestClientSsl.test_ssl[exec_option: {'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0,
> 'num_nodes': 0} | table_format: text/none]
> 08:18:51
> 08:18:51 self =
> 08:18:51 vector =
> 08:18:51
> 08:18:51 @pytest.mark.execute_serially
> 08:18:51
> @CustomClusterTestSuite.with_args("--ssl_server_certificate=%s/server-cert.pem
> "
> 08:18:51
> "--ssl_private_key=%s/server-key.pem"
> 08:18:51 % (CERT_DIR, CERT_DIR))
> 08:18:51 def test_ssl(self, vector):
> 08:18:51
> 08:18:51 self._verify_negative_cases()
> 08:18:51 # TODO: This is really two different tests, but the custom
> cluster takes too long to
> 08:18:51 # start. Make it so that custom clusters can be specified
> across test suites.
> 08:18:51 self._validate_positive_cases("%s/server-cert.pem" %
> self.CERT_DIR)
> 08:18:51
> 08:18:51 # No certificate checking: will accept any cert.
> 08:18:51 self._validate_positive_cases()
> 08:18:51
> 08:18:51 # Test cancelling a query
> 08:18:51 impalad = ImpaladService(socket.getfqdn())
> 08:18:51 impalad.wait_for_num_in_flight_queries(0)
> 08:18:51 p = ImpalaShell(args="--ssl")
> 08:18:51 p.send_cmd("SET DEBUG_ACTION=0:OPEN:WAIT")
> 08:18:51 p.send_cmd("select count(*) from functional.alltypes")
> 08:18:51 impalad.wait_for_num_in_flight_queries(1)
> 08:18:51
> 08:18:51 LOG = logging.getLogger('test_client_ssl')
> 08:18:51 LOG.info("Cancelling query")
> 08:18:51 num_tries = 0
> 08:18:51 # In practice, sending SIGINT to the shell process doesn't
> always seem to get caught
> 08:18:51 # (and a search shows up some bugs in Python where SIGINT
> might be ignored). So retry
> 08:18:51 # for 30s until one signal takes.
> 08:18:51 while impalad.get_num_in_flight_queries() == 1:
> 08:18:51 time.sleep(1)
> 08:18:51 LOG.info("Sending signal...")
> 08:18:51 os.kill(p.pid(), signal.SIGINT)
> 08:18:51 num_tries += 1
> 08:18:51 > assert num_tries < 30, "SIGINT was not caught by shell
> within 30s"
> 08:18:51 E AssertionError: SIGINT was not caught by shell within 30s
> 08:18:51 E assert 30 < 30
> 08:18:51
> 08:18:51 custom_cluster/test_client_ssl.py:85: AssertionError
> 08:18:51 Captured stdout setup
> -
> 08:18:51 Starting State Store logging to
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 08:18:51 Starting Catalog Service logging to
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 08:18:51 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 08:18:51 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 08:18:51 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 08:18:51 Waiting for Catalog... Status: 53 DBs / 1077 tables (ready=True)
> 08:18:51 Waiting for Catalog... Status: 53 DBs / 1077 tables (ready=True)
> 08:18:51 Waiting for Catalog... Status: 53 DBs / 1077 tables (ready=True)
> 08:18:51 Impala Cluster Running with 3 nodes.
> 08:18:51 Captured stderr setup
> -
> 08:18:51
[jira] [Resolved] (IMPALA-10312) test_shell_interactive.TestImpalaShellInteractive AssertionError: alter query should be closed
[
https://issues.apache.org/jira/browse/IMPALA-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10312.
Fix Version/s: Impala 4.0
Resolution: Fixed
> test_shell_interactive.TestImpalaShellInteractive AssertionError: alter query
> should be closed
> --
>
> Key: IMPALA-10312
> URL: https://issues.apache.org/jira/browse/IMPALA-10312
> Project: IMPALA
> Issue Type: Bug
>Reporter: Kurt Deschler
>Assignee: Tim Armstrong
>Priority: Major
> Labels: broken-build, flaky
> Fix For: Impala 4.0
>
>
> shell.test_shell_interactive.TestImpalaShellInteractive.test_ddl_queries_are_closed[table_format_and_file_extension:
> ('textfile', '.txt') | protocol: beeswax] (from pytest)
> Failing for the past 1 build (Since
> [!https://master-02.jenkins.cloudera.com/static/4975bb2b/images/16x16/red.png!
>
> #196|https://master-02.jenkins.cloudera.com/view/Impala/view/Evergreen-asf-master/job/impala-asf-master-core-s3-data-cache/lastCompletedBuild/]
> )
> [Took 22
> sec.|https://master-02.jenkins.cloudera.com/view/Impala/view/Evergreen-asf-master/job/impala-asf-master-core-s3-data-cache/lastCompletedBuild/testReport/shell.test_shell_interactive/TestImpalaShellInteractive/test_ddl_queries_are_closed_table_format_and_file_extensiontextfile_txt_protocol__beeswax_/history]
>
> [!https://master-02.jenkins.cloudera.com/static/4975bb2b/images/16x16/notepad.png!
> add
> description|https://master-02.jenkins.cloudera.com/view/Impala/view/Evergreen-asf-master/job/impala-asf-master-core-s3-data-cache/lastCompletedBuild/testReport/shell.test_shell_interactive/TestImpalaShellInteractive/test_ddl_queries_are_closed_table_format_and_file_extensiontextfile_txt_protocol__beeswax_/editDescription]
> h3. Error Message
> AssertionError: alter query should be closed assert ImpaladService.wait_for_num_in_flight_queries of
> >(0) +
> where > =
> 0x7f49541b2150>.wait_for_num_in_flight_queries
> h3. Stacktrace
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/tests/shell/test_shell_interactive.py:467:
> in test_ddl_queries_are_closed assert
> impalad.wait_for_num_in_flight_queries(0), MSG % 'alter' E AssertionError:
> alter query should be closed E assert ImpaladService.wait_for_num_in_flight_queries of
> >(0) E +
> where > =
> 0x7f49541b2150>.wait_for_num_in_flight_queries
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10332) Add file formats to HdfsScanNode's thrift representation and codegen for those
[ https://issues.apache.org/jira/browse/IMPALA-10332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-10332: --- Fix Version/s: Impala 4.0 > Add file formats to HdfsScanNode's thrift representation and codegen for those > -- > > Key: IMPALA-10332 > URL: https://issues.apache.org/jira/browse/IMPALA-10332 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Fix For: Impala 4.0 > > > List all file formats that a HdfsScanNode needs to process in any fragment > instance. It is possible that some file formats will not be needed in all > fragment instances. > This is a step towards sharing codegen between different impala backends. > Using the file formats provided in the thrift file, a backend can codegen > code for file formats that are not needed in its own process but are needed > in other fragment instances running on other backends, and the resulting > binary can be shared between multiple backends. > Codegenning for file formats will be done based on the thrift message and not > on what is needed for the actual backend. This leads to some extra work in > case a file format is not needed for the current backend and codegen sharing > is not available (at this point it is not implemented). However, the overall > number of such cases is low. > Also adding the file formats to the node's explain string. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10216) BufferPoolTest.WriteErrorBlacklistCompression is flaky on UBSAN builds
[
https://issues.apache.org/jira/browse/IMPALA-10216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237757#comment-17237757
]
Tim Armstrong commented on IMPALA-10216:
[~fangyurao] can you also include the LastTest/LastTestFailed log files?
> BufferPoolTest.WriteErrorBlacklistCompression is flaky on UBSAN builds
> --
>
> Key: IMPALA-10216
> URL: https://issues.apache.org/jira/browse/IMPALA-10216
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: flaky
> Attachments: buffer-pool-test.ERROR, buffer-pool-test.INFO,
> buffer-pool-test.WARNING, buffer-pool-test.xml
>
>
> Only seen this once so far:
> {code}
> BufferPoolTest.WriteErrorBlacklistCompression
> Error Message
> Value of: FindPageInDir(pages[NO_ERROR_QUERY], error_dir) != NULL
> Actual: false
> Expected: true
> Stacktrace
> Impala/be/src/runtime/bufferpool/buffer-pool-test.cc:1764
> Value of: FindPageInDir(pages[NO_ERROR_QUERY], error_dir) != NULL
> Actual: false
> Expected: true
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9566) Sentry service should not be started after IMPALA-8870
[
https://issues.apache.org/jira/browse/IMPALA-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-9566.
---
Fix Version/s: Impala 4.0
Resolution: Fixed
> Sentry service should not be started after IMPALA-8870
> --
>
> Key: IMPALA-9566
> URL: https://issues.apache.org/jira/browse/IMPALA-9566
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend, Infrastructure
>Reporter: Fang-Yu Rao
>Assignee: Fang-Yu Rao
>Priority: Major
> Labels: broken-build
> Fix For: Impala 4.0
>
>
> Due to the incompatibility between different versions of the Guava libraries.
> After bumping up Guava in IMPALA-8870, our build script is not supposed to
> start up the Sentry service because Sentry does not have its Guava bumped up
> yet.
> To fix this, when starting the underlying services, we should take into
> consideration the case when {{$\{DEFAULT_FS}}} is neither
> "{{hdfs://${INTERNAL_LISTEN_HOST}:20500}}" nor "{{${LOCAL_FS}}}".
> Specifically, we should not attempt to start the Sentry server in the
> following case.
> [https://github.com/apache/impala/blob/master/testdata/bin/run-all.sh#L102-L104]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-9550) TestResultSpoolingFetchSize.test_fetch is flaky
[
https://issues.apache.org/jira/browse/IMPALA-9550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-9550:
-
Assignee: Riza Suminto
> TestResultSpoolingFetchSize.test_fetch is flaky
> ---
>
> Key: IMPALA-9550
> URL: https://issues.apache.org/jira/browse/IMPALA-9550
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Riza Suminto
>Priority: Major
> Labels: broken-build, flaky
>
> Looks like the timeout needs to be bumped up.
> {code}
> query_test.test_result_spooling.TestResultSpoolingFetchSize.test_fetch[fetch_size:
> 1 | protocol: beeswax | exec_option: {'batch_size': 2048, 'num_nodes':
> 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none | wait_for_finished: True] (from pytest)
> Error Message
> query_test/test_result_spooling.py:292: in test_fetch
> self.wait_for_state(handle, self.client.QUERY_STATES['FINISHED'], timeout)
> common/impala_test_suite.py:1085: in wait_for_state
> self.wait_for_any_state(handle, [expected_state], timeout, client)
> common/impala_test_suite.py:1102: in wait_for_any_state actual_state)) E
> Timeout: query 424f02e1ff0912bc:37a5e6cb did not reach one of the
> expected states [4], last known state 2
> Stacktrace
> query_test/test_result_spooling.py:292: in test_fetch
> self.wait_for_state(handle, self.client.QUERY_STATES['FINISHED'], timeout)
> common/impala_test_suite.py:1085: in wait_for_state
> self.wait_for_any_state(handle, [expected_state], timeout, client)
> common/impala_test_suite.py:1102: in wait_for_any_state
> actual_state))
> E Timeout: query 424f02e1ff0912bc:37a5e6cb did not reach one of the
> expected states [4], last known state 2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10351) Enable mt_dop for DML
Tim Armstrong created IMPALA-10351: -- Summary: Enable mt_dop for DML Key: IMPALA-10351 URL: https://issues.apache.org/jira/browse/IMPALA-10351 Project: IMPALA Issue Type: Improvement Components: Frontend Reporter: Tim Armstrong Assignee: Tim Armstrong We don't have any pending issues for MT_DOP with DML and have sufficient functional testing, so I think we can go ahead and enable it. IMPALA-8125 and query hints give us a safety valve in case we overparallelize. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9588) test_cancel_insert failed with ImpalaBeeswaxException
[
https://issues.apache.org/jira/browse/IMPALA-9588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237683#comment-17237683
]
Tim Armstrong commented on IMPALA-9588:
---
[~gaborkaszab] could you maybe take a quick look at this. I don't think we have
enough logs to root cause this, but the logging in the test seems inadequate.
AFAICT the thread running __fetch_results hit an error, but that error is not
logged anywhere by the tests.
https://github.com/apache/impala/blob/86e611bdb2a912e283ce1db01bf2940ee1583ff3/tests/util/cancel_util.py#L24
I think we probably just need more logging to understand what happened.
> test_cancel_insert failed with ImpalaBeeswaxException
> -
>
> Key: IMPALA-9588
> URL: https://issues.apache.org/jira/browse/IMPALA-9588
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Yongzhi Chen
>Assignee: Gabor Kaszab
>Priority: Major
> Labels: broken-build, flaky
>
> test_cancellation.TestCancellationSerial.test_cancel_insert failed in
> impala-asf-master-core-s3 build:
> query_test.test_cancellation.TestCancellationSerial.test_cancel_insert[protocol:
> beeswax | table_format: parquet/none | exec_option: {'batch_size': 0,
> 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen':
> False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} |
> query_type: CTAS | mt_dop: 0 | wait_action: None | cancel_delay: 1 |
> cpu_limit_s: 0 | query: select * from lineitem order by l_orderkey |
> fail_rpc_action: None | join_before_close: False | buffer_pool_limit: 0]
> Error Message
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: 'thrift.Thrift.TApplicationException'> MESSAGE: TException - service has
> thrown: BeeswaxException(message=Invalid query handle:
> cc4c5258e88c790b:8db6f87d, log_context=, handle=QueryHandle(id=,
> log_context=), errorCode=0, SQLState=HY000)
> Stacktrace
> query_test/test_cancellation.py:248: in test_cancel_insert
> self.execute_cancel_test(vector)
> query_test/test_cancellation.py:167: in execute_cancel_test
> vector.get_value('cancel_delay'), vector.get_value('join_before_close'))
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/util/cancel_util.py:41:
> in cancel_query_and_validate_state
> assert client.get_state(handle) != client.QUERY_STATES['EXCEPTION']
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_connection.py:219:
> in get_state
> return self.__beeswax_client.get_state(operation_handle.get_handle())
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py:433:
> in get_state
> return self.__do_rpc(lambda: self.imp_service.get_state(query_handle))
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py:525:
> in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(t), t)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: TException - service has thrown:
> BeeswaxException(message=Invalid query handle:
> cc4c5258e88c790b:8db6f87d, log_context=, handle=QueryHandle(id=,
> log_context=), errorCode=0, SQLState=HY000)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8125) Limit number of files generated by insert
[ https://issues.apache.org/jira/browse/IMPALA-8125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-8125. --- Fix Version/s: Impala 4.0 Resolution: Fixed > Limit number of files generated by insert > - > > Key: IMPALA-8125 > URL: https://issues.apache.org/jira/browse/IMPALA-8125 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Reporter: Tim Armstrong >Assignee: Bikramjeet Vig >Priority: Major > Labels: multithreading > Fix For: Impala 4.0 > > > One pitfall of multithreaded execution is that, if implemented naively, the > number of files generated by an unpartitioned insert will be multiplied by > mt_dop. > We should provide a mechanism to limit the number of files generated, e.g. > limit the number of insert fragment instances (note that there a pre-existing > problem with unpartitioned inserts generating too many files). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-9588) test_cancel_insert failed with ImpalaBeeswaxException
[
https://issues.apache.org/jira/browse/IMPALA-9588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-9588:
-
Assignee: Gabor Kaszab
> test_cancel_insert failed with ImpalaBeeswaxException
> -
>
> Key: IMPALA-9588
> URL: https://issues.apache.org/jira/browse/IMPALA-9588
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Yongzhi Chen
>Assignee: Gabor Kaszab
>Priority: Major
> Labels: broken-build, flaky
>
> test_cancellation.TestCancellationSerial.test_cancel_insert failed in
> impala-asf-master-core-s3 build:
> query_test.test_cancellation.TestCancellationSerial.test_cancel_insert[protocol:
> beeswax | table_format: parquet/none | exec_option: {'batch_size': 0,
> 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen':
> False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} |
> query_type: CTAS | mt_dop: 0 | wait_action: None | cancel_delay: 1 |
> cpu_limit_s: 0 | query: select * from lineitem order by l_orderkey |
> fail_rpc_action: None | join_before_close: False | buffer_pool_limit: 0]
> Error Message
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: 'thrift.Thrift.TApplicationException'> MESSAGE: TException - service has
> thrown: BeeswaxException(message=Invalid query handle:
> cc4c5258e88c790b:8db6f87d, log_context=, handle=QueryHandle(id=,
> log_context=), errorCode=0, SQLState=HY000)
> Stacktrace
> query_test/test_cancellation.py:248: in test_cancel_insert
> self.execute_cancel_test(vector)
> query_test/test_cancellation.py:167: in execute_cancel_test
> vector.get_value('cancel_delay'), vector.get_value('join_before_close'))
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/util/cancel_util.py:41:
> in cancel_query_and_validate_state
> assert client.get_state(handle) != client.QUERY_STATES['EXCEPTION']
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_connection.py:219:
> in get_state
> return self.__beeswax_client.get_state(operation_handle.get_handle())
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py:433:
> in get_state
> return self.__do_rpc(lambda: self.imp_service.get_state(query_handle))
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py:525:
> in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(t), t)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: TException - service has thrown:
> BeeswaxException(message=Invalid query handle:
> cc4c5258e88c790b:8db6f87d, log_context=, handle=QueryHandle(id=,
> log_context=), errorCode=0, SQLState=HY000)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9465) TotalReadThroughput is sporadically not set in profile
[ https://issues.apache.org/jira/browse/IMPALA-9465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-9465: -- Labels: ramp-up (was: flaky ramp-up) > TotalReadThroughput is sporadically not set in profile > -- > > Key: IMPALA-9465 > URL: https://issues.apache.org/jira/browse/IMPALA-9465 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Csaba Ringhofer >Priority: Minor > Labels: ramp-up > > I saw "TotalReadThroughput: 0.00 /sec" several times in profiles locally > while BytesRead is not 0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8452) Avro scanner seems broken
[
https://issues.apache.org/jira/browse/IMPALA-8452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-8452.
---
Resolution: Cannot Reproduce
I can no longer repro the issue with test_tpch_scan_ranges
> Avro scanner seems broken
> -
>
> Key: IMPALA-8452
> URL: https://issues.apache.org/jira/browse/IMPALA-8452
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.3.0
>Reporter: Michael Ho
>Priority: Major
> Labels: broken-build, wrongresults
>
> A few scanner tests started failing recently on Centos6. Coincidentally, both
> of them only started happening after [this
> commit|https://github.com/apache/impala/commit/b5805de3e65fd1c7154e4169b323bb38ddc54f4f].
> [~attilaj], can you please take a look and reassign if you think that commit
> is unrelated ?
> Oddly enough, this has shown up on Centos6. Other exhaustive runs with
> Centos7 seem to work fine. May be it's related to some platform's library ?
> In the first case, a select count star from an avro table hangs for 2 hours:
> {noformat}
> query_test/test_scanners_fuzz.py:83: in test_fuzz_alltypes
> self.run_fuzz_test(vector, src_db, table_name, unique_database,
> table_name)
> query_test/test_scanners_fuzz.py:201: in run_fuzz_test
> result = self.execute_query(query, query_options = query_options)
> common/impala_test_suite.py:619: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:650: in execute_query
> return self.__execute_query(self.client, query, query_options)
> common/impala_test_suite.py:721: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:180: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:183: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:360: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:384: in wait_for_finished
> time.sleep(0.05)
> E Failed: Timeout >7200s
> SET
> client_identifier=query_test/test_scanners_fuzz.py::TestScannersFuzzing::()::test_fuzz_alltypes[protocol:beeswax|exec_option:{'debug_action':None;'abort_on_error':False;'mem_limit':'512m';'num_nodes':0}|table_format:avro/none];
> SET batch_size=1;
> SET num_nodes=0;
> SET disable_codegen_rows_threshold=0;
> SET disable_codegen=True;
> SET abort_on_error=False;
> SET mem_limit=512m;
> -- executing against localhost:21000
> select count(*) from test_fuzz_alltypes_2cdcb963.alltypes q;
> -- 2019-04-24 04:14:31,857 INFO MainThread: Started query
> 2049069f9f5e3aa8:f2fd47ff
> {noformat}
> The second case has to do with incorrect number of rows in a select count
> star from tpch_avro.lineitem:
> {noformat}
> query_test/test_scanners.py:947: in test_tpch_scan_ranges
> self.run_test_case('tpch-scan-range-lengths', vector)
> common/impala_test_suite.py:517: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:370: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:449: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:271: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 6001215 != 6000679
> -- 2019-04-24 03:43:42,805 INFO MainThread: max_scan_range_length=8412307
> SET
> client_identifier=query_test/test_scanners.py::TestTpchScanRangeLengths::()::test_tpch_scan_ranges[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshold':0}|;
> -- executing against localhost:21000
> use tpch_avro;
> -- 2019-04-24 03:43:42,814 INFO MainThread: Started query
> c04e1968443b52fc:5b99b1b3
> SET
> client_identifier=query_test/test_scanners.py::TestTpchScanRangeLengths::()::test_tpch_scan_ranges[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshold':0}|;
> SET batch_size=0;
> SET num_nodes=0;
> SET disable_codegen_rows_threshold=0;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET max_scan_range_length=8412307;
> SET exec_single_node_rows_threshold=0;
> -- executing against localhost:21000
> select count(*)
> from lineitem;
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands,
[jira] [Commented] (IMPALA-10066) test_cancellation_mid_command fails in parallel-all-tests-nightly
[
https://issues.apache.org/jira/browse/IMPALA-10066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237671#comment-17237671
]
Tim Armstrong commented on IMPALA-10066:
Seems like the test failed in precommit, which is not a good sign.
> test_cancellation_mid_command fails in parallel-all-tests-nightly
> -
>
> Key: IMPALA-10066
> URL: https://issues.apache.org/jira/browse/IMPALA-10066
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Gabor Kaszab
>Priority: Major
> Labels: broken-build
>
> Failed jobs:
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11609/]
> [https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/2927/testReport/]
> Stack trace:
> {noformat}
> shell/test_shell_interactive.py:326: in test_cancellation_mid_command
> child_proc.wait()
> ../infra/python/env-gcc7.5.0/lib/python2.7/site-packages/pexpect/__init__.py:1205:
> in wait
> raise ExceptionPexpect('Cannot wait for dead child process.')
> E ExceptionPexpect: Cannot wait for dead child process.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-10066) test_cancellation_mid_command fails in parallel-all-tests-nightly
[
https://issues.apache.org/jira/browse/IMPALA-10066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-10066:
--
Assignee: Gabor Kaszab (was: Adam Tamas)
> test_cancellation_mid_command fails in parallel-all-tests-nightly
> -
>
> Key: IMPALA-10066
> URL: https://issues.apache.org/jira/browse/IMPALA-10066
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Gabor Kaszab
>Priority: Major
> Labels: broken-build
>
> Failed jobs:
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11609/]
> [https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/2927/testReport/]
> Stack trace:
> {noformat}
> shell/test_shell_interactive.py:326: in test_cancellation_mid_command
> child_proc.wait()
> ../infra/python/env-gcc7.5.0/lib/python2.7/site-packages/pexpect/__init__.py:1205:
> in wait
> raise ExceptionPexpect('Cannot wait for dead child process.')
> E ExceptionPexpect: Cannot wait for dead child process.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work stopped] (IMPALA-10066) test_cancellation_mid_command fails in parallel-all-tests-nightly
[
https://issues.apache.org/jira/browse/IMPALA-10066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-10066 stopped by Tim Armstrong.
--
> test_cancellation_mid_command fails in parallel-all-tests-nightly
> -
>
> Key: IMPALA-10066
> URL: https://issues.apache.org/jira/browse/IMPALA-10066
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Adam Tamas
>Priority: Major
> Labels: broken-build
>
> Failed jobs:
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11609/]
> [https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/2927/testReport/]
> Stack trace:
> {noformat}
> shell/test_shell_interactive.py:326: in test_cancellation_mid_command
> child_proc.wait()
> ../infra/python/env-gcc7.5.0/lib/python2.7/site-packages/pexpect/__init__.py:1205:
> in wait
> raise ExceptionPexpect('Cannot wait for dead child process.')
> E ExceptionPexpect: Cannot wait for dead child process.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10350) Impala loses double precision because of DECIMAL->DOUBLE cast
[
https://issues.apache.org/jira/browse/IMPALA-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237668#comment-17237668
]
Tim Armstrong commented on IMPALA-10350:
You need a specialized algorithm (like used in something like strtof) to do
this accurately. A dumb but correct solution would be to convert the decimal to
a string, then parse the string. I found a couple of high performance examples
of the core algorithm with Apache licenses.
https://github.com/lemire/fast_double_parser/blob/master/include/fast_double_parser.h#L884
https://github.com/google/wuffs/blob/7ea23d56e3fe9e4adff95ebf11bc18b9cb06e0d5/internal/cgen/base/floatconv-submodule-code.c#L1262
Note that both of these implement float parsing, but do it by first converting
to a representation similar to Impala's DecimalValue - i.e. an integer plus a
scale, then converting to double.
> Impala loses double precision because of DECIMAL->DOUBLE cast
> -
>
> Key: IMPALA-10350
> URL: https://issues.apache.org/jira/browse/IMPALA-10350
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Zoltán Borók-Nagy
>Priority: Major
> Labels: correctness, ramp-up
> Attachments: test.c
>
>
> Impala might loses presision of double values. Reproduction:
> {noformat}
> create table double_tbl (d double) stored as textfile;
> insert into double_tbl values (-0.43149576573887316);
> {noformat}
> Then inspect the data file:
> {noformat}
> $ hdfs dfs -cat
> /test-warehouse/double_tbl/424097c644088674-c55b9101_175064830_data.0.txt
> -0.4314957657388731{noformat}
> The same happens if we store our data in Parquet.
> Hive writes don't lose precision. If the data was written by Hive then Impala
> can read the values correctly:
> {noformat}
> $ bin/run-jdbc-client.sh -t NOSASL -q "select * from double_tbl;"
> Using JDBC Driver Name: org.apache.hive.jdbc.HiveDriver
> Connecting to: jdbc:hive2://localhost:21050/;auth=noSasl
> Executing: select * from double_tbl
> [START]
> -0.43149576573887316
> [END]{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10350) Impala loses double precision because of DECIMAL->DOUBLE cast
[
https://issues.apache.org/jira/browse/IMPALA-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-10350:
---
Summary: Impala loses double precision because of DECIMAL->DOUBLE cast
(was: Impala loses double precision on the write side)
> Impala loses double precision because of DECIMAL->DOUBLE cast
> -
>
> Key: IMPALA-10350
> URL: https://issues.apache.org/jira/browse/IMPALA-10350
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Zoltán Borók-Nagy
>Priority: Major
> Labels: correctness, ramp-up
> Attachments: test.c
>
>
> Impala might loses presision of double values. Reproduction:
> {noformat}
> create table double_tbl (d double) stored as textfile;
> insert into double_tbl values (-0.43149576573887316);
> {noformat}
> Then inspect the data file:
> {noformat}
> $ hdfs dfs -cat
> /test-warehouse/double_tbl/424097c644088674-c55b9101_175064830_data.0.txt
> -0.4314957657388731{noformat}
> The same happens if we store our data in Parquet.
> Hive writes don't lose precision. If the data was written by Hive then Impala
> can read the values correctly:
> {noformat}
> $ bin/run-jdbc-client.sh -t NOSASL -q "select * from double_tbl;"
> Using JDBC Driver Name: org.apache.hive.jdbc.HiveDriver
> Connecting to: jdbc:hive2://localhost:21050/;auth=noSasl
> Executing: select * from double_tbl
> [START]
> -0.43149576573887316
> [END]{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10350) Impala loses double precision on the write side
[
https://issues.apache.org/jira/browse/IMPALA-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-10350:
---
Labels: correctness ramp-up (was: correctness)
> Impala loses double precision on the write side
> ---
>
> Key: IMPALA-10350
> URL: https://issues.apache.org/jira/browse/IMPALA-10350
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Zoltán Borók-Nagy
>Priority: Major
> Labels: correctness, ramp-up
> Attachments: test.c
>
>
> Impala might loses presision of double values. Reproduction:
> {noformat}
> create table double_tbl (d double) stored as textfile;
> insert into double_tbl values (-0.43149576573887316);
> {noformat}
> Then inspect the data file:
> {noformat}
> $ hdfs dfs -cat
> /test-warehouse/double_tbl/424097c644088674-c55b9101_175064830_data.0.txt
> -0.4314957657388731{noformat}
> The same happens if we store our data in Parquet.
> Hive writes don't lose precision. If the data was written by Hive then Impala
> can read the values correctly:
> {noformat}
> $ bin/run-jdbc-client.sh -t NOSASL -q "select * from double_tbl;"
> Using JDBC Driver Name: org.apache.hive.jdbc.HiveDriver
> Connecting to: jdbc:hive2://localhost:21050/;auth=noSasl
> Executing: select * from double_tbl
> [START]
> -0.43149576573887316
> [END]{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10350) Impala loses double precision on the write side
[
https://issues.apache.org/jira/browse/IMPALA-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237560#comment-17237560
]
Tim Armstrong commented on IMPALA-10350:
I think the problem is this function which is used in the DECIMAL -> DOUBLE
cast:
https://github.com/apache/impala/blob/45c105d/be/src/runtime/decimal-value.inline.h#L725
Here's a small test program indicating that it seems to be where the truncation
happens.
[^test.c]
{noformat}
$ gcc test.c -o test -lm && ./test
-0.43149576573887316 -0.4314957657390
-0.43149576573887310
{noformat}
> Impala loses double precision on the write side
> ---
>
> Key: IMPALA-10350
> URL: https://issues.apache.org/jira/browse/IMPALA-10350
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Zoltán Borók-Nagy
>Priority: Major
> Labels: correctness
> Attachments: test.c
>
>
> Impala might loses presision of double values. Reproduction:
> {noformat}
> create table double_tbl (d double) stored as textfile;
> insert into double_tbl values (-0.43149576573887316);
> {noformat}
> Then inspect the data file:
> {noformat}
> $ hdfs dfs -cat
> /test-warehouse/double_tbl/424097c644088674-c55b9101_175064830_data.0.txt
> -0.4314957657388731{noformat}
> The same happens if we store our data in Parquet.
> Hive writes don't lose precision. If the data was written by Hive then Impala
> can read the values correctly:
> {noformat}
> $ bin/run-jdbc-client.sh -t NOSASL -q "select * from double_tbl;"
> Using JDBC Driver Name: org.apache.hive.jdbc.HiveDriver
> Connecting to: jdbc:hive2://localhost:21050/;auth=noSasl
> Executing: select * from double_tbl
> [START]
> -0.43149576573887316
> [END]{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10350) Impala loses double precision on the write side
[
https://issues.apache.org/jira/browse/IMPALA-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-10350:
---
Attachment: test.c
> Impala loses double precision on the write side
> ---
>
> Key: IMPALA-10350
> URL: https://issues.apache.org/jira/browse/IMPALA-10350
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Zoltán Borók-Nagy
>Priority: Major
> Labels: correctness
> Attachments: test.c
>
>
> Impala might loses presision of double values. Reproduction:
> {noformat}
> create table double_tbl (d double) stored as textfile;
> insert into double_tbl values (-0.43149576573887316);
> {noformat}
> Then inspect the data file:
> {noformat}
> $ hdfs dfs -cat
> /test-warehouse/double_tbl/424097c644088674-c55b9101_175064830_data.0.txt
> -0.4314957657388731{noformat}
> The same happens if we store our data in Parquet.
> Hive writes don't lose precision. If the data was written by Hive then Impala
> can read the values correctly:
> {noformat}
> $ bin/run-jdbc-client.sh -t NOSASL -q "select * from double_tbl;"
> Using JDBC Driver Name: org.apache.hive.jdbc.HiveDriver
> Connecting to: jdbc:hive2://localhost:21050/;auth=noSasl
> Executing: select * from double_tbl
> [START]
> -0.43149576573887316
> [END]{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10350) Impala loses double precision on the write side
[
https://issues.apache.org/jira/browse/IMPALA-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-10350:
---
Attachment: test.c
> Impala loses double precision on the write side
> ---
>
> Key: IMPALA-10350
> URL: https://issues.apache.org/jira/browse/IMPALA-10350
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Zoltán Borók-Nagy
>Priority: Major
> Labels: correctness
>
> Impala might loses presision of double values. Reproduction:
> {noformat}
> create table double_tbl (d double) stored as textfile;
> insert into double_tbl values (-0.43149576573887316);
> {noformat}
> Then inspect the data file:
> {noformat}
> $ hdfs dfs -cat
> /test-warehouse/double_tbl/424097c644088674-c55b9101_175064830_data.0.txt
> -0.4314957657388731{noformat}
> The same happens if we store our data in Parquet.
> Hive writes don't lose precision. If the data was written by Hive then Impala
> can read the values correctly:
> {noformat}
> $ bin/run-jdbc-client.sh -t NOSASL -q "select * from double_tbl;"
> Using JDBC Driver Name: org.apache.hive.jdbc.HiveDriver
> Connecting to: jdbc:hive2://localhost:21050/;auth=noSasl
> Executing: select * from double_tbl
> [START]
> -0.43149576573887316
> [END]{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10350) Impala loses double precision on the write side
[
https://issues.apache.org/jira/browse/IMPALA-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-10350:
---
Attachment: (was: test.c)
> Impala loses double precision on the write side
> ---
>
> Key: IMPALA-10350
> URL: https://issues.apache.org/jira/browse/IMPALA-10350
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Zoltán Borók-Nagy
>Priority: Major
> Labels: correctness
>
> Impala might loses presision of double values. Reproduction:
> {noformat}
> create table double_tbl (d double) stored as textfile;
> insert into double_tbl values (-0.43149576573887316);
> {noformat}
> Then inspect the data file:
> {noformat}
> $ hdfs dfs -cat
> /test-warehouse/double_tbl/424097c644088674-c55b9101_175064830_data.0.txt
> -0.4314957657388731{noformat}
> The same happens if we store our data in Parquet.
> Hive writes don't lose precision. If the data was written by Hive then Impala
> can read the values correctly:
> {noformat}
> $ bin/run-jdbc-client.sh -t NOSASL -q "select * from double_tbl;"
> Using JDBC Driver Name: org.apache.hive.jdbc.HiveDriver
> Connecting to: jdbc:hive2://localhost:21050/;auth=noSasl
> Executing: select * from double_tbl
> [START]
> -0.43149576573887316
> [END]{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10350) Impala loses double precision on the write side
[
https://issues.apache.org/jira/browse/IMPALA-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237555#comment-17237555
]
Tim Armstrong commented on IMPALA-10350:
The above was slightly misleading since it returned a DECIMAL value. If you
cast to double, you get the same value as you get from reading from the data
file.
{noformat}
[localhost:21050] default> select cast(-0.43149576573887316 as double);
Query: select cast(-0.43149576573887316 as double)
Query submitted at: 2020-11-23 09:49:32 (Coordinator:
http://tarmstrong-box:25000)
Query progress can be monitored at:
http://tarmstrong-box:25000/query_plan?query_id=6c414c04b47d0897:10bb3cc5
+--+
| cast(-0.43149576573887316 as double) |
+--+
| -0.431495765739 |
+--+
{noformat}
The expressions are
{noformat}
[0] = TExprNode {
01: node_type (i32) = 14,
02: type (struct) = TColumnType {
01: types (list) = list[1] {
[0] = TTypeNode {
01: type (i32) = 0,
02: scalar_type (struct) = TScalarType {
01: type (i32) = 8,
},
},
},
},
03: num_children (i32) = 1,
04: is_constant (bool) = false,
05: fn (struct) = TFunction {
01: name (struct) = TFunctionName {
01: db_name (string) = "_impala_builtins",
02: function_name (string) = "casttodouble",
},
02: binary_type (i32) = 0,
03: arg_types (list) = list[1] {
[0] = TColumnType {
01: types (list) = list[1] {
[0] = TTypeNode {
01: type (i32) = 0,
02: scalar_type (struct) = TScalarType {
01: type (i32) = 14,
03: precision (i32) = -1,
04: scale (i32) = -1,
},
},
},
},
},
...
[0] = TExpr {
01: nodes (list) = list[1] {
[0] = TExprNode {
01: node_type (i32) = 5,
02: type (struct) = TColumnType {
01: types (list) = list[1] {
[0] = TTypeNode {
01: type (i32) = 0,
02: scalar_type (struct) = TScalarType {
01: type (i32) = 14,
03: precision (i32) = 17,
04: scale (i32) = 17,
},
},
},
},
03: num_children (i32) = 0,
04: is_constant (bool) = true,
18: decimal_literal (struct) = TDecimalLiteral {
01: value (string) =
"\xfff\xb3\xb0P\x1a\xdc\xac",
},
},
},
{noformat}
> Impala loses double precision on the write side
> ---
>
> Key: IMPALA-10350
> URL: https://issues.apache.org/jira/browse/IMPALA-10350
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Zoltán Borók-Nagy
>Priority: Major
> Labels: correctness
>
> Impala might loses presision of double values. Reproduction:
> {noformat}
> create table double_tbl (d double) stored as textfile;
> insert into double_tbl values (-0.43149576573887316);
> {noformat}
> Then inspect the data file:
> {noformat}
> $ hdfs dfs -cat
> /test-warehouse/double_tbl/424097c644088674-c55b9101_175064830_data.0.txt
> -0.4314957657388731{noformat}
> The same happens if we store our data in Parquet.
> Hive writes don't lose precision. If the data was written by Hive then Impala
> can read the values correctly:
> {noformat}
> $ bin/run-jdbc-client.sh -t NOSASL -q "select * from double_tbl;"
> Using JDBC Driver Name: org.apache.hive.jdbc.HiveDriver
> Connecting to: jdbc:hive2://localhost:21050/;auth=noSasl
> Executing:
[jira] [Commented] (IMPALA-10350) Impala loses double precision on the write side
[
https://issues.apache.org/jira/browse/IMPALA-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237541#comment-17237541
]
Tim Armstrong commented on IMPALA-10350:
There does seem to be something going on here beyond just usual floating point
rounding stuff, since if we just do a select a different value comes back.
{noformat}
[localhost:21050] default> select -0.43149576573887316;
Query: select -0.43149576573887316
Query submitted at: 2020-11-23 09:35:33 (Coordinator:
http://tarmstrong-box:25000)
Query progress can be monitored at:
http://tarmstrong-box:25000/query_plan?query_id=044e2e614cfd3791:dc64f9cd
+--+
| -0.43149576573887316 |
+--+
| -0.43149576573887316 |
+--+
{noformat}
> Impala loses double precision on the write side
> ---
>
> Key: IMPALA-10350
> URL: https://issues.apache.org/jira/browse/IMPALA-10350
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Zoltán Borók-Nagy
>Priority: Major
> Labels: correctness
>
> Impala might loses presision of double values. Reproduction:
> {noformat}
> create table double_tbl (d double) stored as textfile;
> insert into double_tbl values (-0.43149576573887316);
> {noformat}
> Then inspect the data file:
> {noformat}
> $ hdfs dfs -cat
> /test-warehouse/double_tbl/424097c644088674-c55b9101_175064830_data.0.txt
> -0.4314957657388731{noformat}
> The same happens if we store our data in Parquet.
> Hive writes don't lose precision. If the data was written by Hive then Impala
> can read the values correctly:
> {noformat}
> $ bin/run-jdbc-client.sh -t NOSASL -q "select * from double_tbl;"
> Using JDBC Driver Name: org.apache.hive.jdbc.HiveDriver
> Connecting to: jdbc:hive2://localhost:21050/;auth=noSasl
> Executing: select * from double_tbl
> [START]
> -0.43149576573887316
> [END]{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10350) Impala loses double precision on the write side
[
https://issues.apache.org/jira/browse/IMPALA-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-10350:
---
Component/s: Backend
> Impala loses double precision on the write side
> ---
>
> Key: IMPALA-10350
> URL: https://issues.apache.org/jira/browse/IMPALA-10350
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Zoltán Borók-Nagy
>Priority: Major
> Labels: correctness
>
> Impala might loses presision of double values. Reproduction:
> {noformat}
> create table double_tbl (d double) stored as textfile;
> insert into double_tbl values (-0.43149576573887316);
> {noformat}
> Then inspect the data file:
> {noformat}
> $ hdfs dfs -cat
> /test-warehouse/double_tbl/424097c644088674-c55b9101_175064830_data.0.txt
> -0.4314957657388731{noformat}
> The same happens if we store our data in Parquet.
> Hive writes don't lose precision. If the data was written by Hive then Impala
> can read the values correctly:
> {noformat}
> $ bin/run-jdbc-client.sh -t NOSASL -q "select * from double_tbl;"
> Using JDBC Driver Name: org.apache.hive.jdbc.HiveDriver
> Connecting to: jdbc:hive2://localhost:21050/;auth=noSasl
> Executing: select * from double_tbl
> [START]
> -0.43149576573887316
> [END]{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10350) Impala loses double precision on the write side
[
https://issues.apache.org/jira/browse/IMPALA-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-10350:
---
Labels: correctness (was: )
> Impala loses double precision on the write side
> ---
>
> Key: IMPALA-10350
> URL: https://issues.apache.org/jira/browse/IMPALA-10350
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Priority: Major
> Labels: correctness
>
> Impala might loses presision of double values. Reproduction:
> {noformat}
> create table double_tbl (d double) stored as textfile;
> insert into double_tbl values (-0.43149576573887316);
> {noformat}
> Then inspect the data file:
> {noformat}
> $ hdfs dfs -cat
> /test-warehouse/double_tbl/424097c644088674-c55b9101_175064830_data.0.txt
> -0.4314957657388731{noformat}
> The same happens if we store our data in Parquet.
> Hive writes don't lose precision. If the data was written by Hive then Impala
> can read the values correctly:
> {noformat}
> $ bin/run-jdbc-client.sh -t NOSASL -q "select * from double_tbl;"
> Using JDBC Driver Name: org.apache.hive.jdbc.HiveDriver
> Connecting to: jdbc:hive2://localhost:21050/;auth=noSasl
> Executing: select * from double_tbl
> [START]
> -0.43149576573887316
> [END]{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work stopped] (IMPALA-9884) TestAdmissionControllerStress.test_mem_limit failing occasionally
[
https://issues.apache.org/jira/browse/IMPALA-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-9884 stopped by Tim Armstrong.
-
> TestAdmissionControllerStress.test_mem_limit failing occasionally
> -
>
> Key: IMPALA-9884
> URL: https://issues.apache.org/jira/browse/IMPALA-9884
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build, flaky
> Attachments: impalad-executors.tar.gz,
> impalad.impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com.jenkins.log.INFO.20201017-06.23933.gz
>
>
> Recently, I saw this test failing with the exception trace below.
> {noformat}
> custom_cluster/test_admission_controller.py:1782: in test_mem_limit
> {'request_pool': self.pool_name, 'mem_limit': query_mem_limit})
> custom_cluster/test_admission_controller.py:1638: in run_admission_test
> assert metric_deltas['dequeued'] == 0,\
> E AssertionError: Queued queries should not run until others are made to
> finish
> E assert 1 == 0
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9884) TestAdmissionControllerStress.test_mem_limit failing occasionally
[
https://issues.apache.org/jira/browse/IMPALA-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-9884 started by Tim Armstrong.
-
> TestAdmissionControllerStress.test_mem_limit failing occasionally
> -
>
> Key: IMPALA-9884
> URL: https://issues.apache.org/jira/browse/IMPALA-9884
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build, flaky
> Attachments: impalad-executors.tar.gz,
> impalad.impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com.jenkins.log.INFO.20201017-06.23933.gz
>
>
> Recently, I saw this test failing with the exception trace below.
> {noformat}
> custom_cluster/test_admission_controller.py:1782: in test_mem_limit
> {'request_pool': self.pool_name, 'mem_limit': query_mem_limit})
> custom_cluster/test_admission_controller.py:1638: in run_admission_test
> assert metric_deltas['dequeued'] == 0,\
> E AssertionError: Queued queries should not run until others are made to
> finish
> E assert 1 == 0
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-10125) webserver.test_web_pages.TestWebPage.test_catalog failed
[
https://issues.apache.org/jira/browse/IMPALA-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-10125:
--
Assignee: (was: Tim Armstrong)
> webserver.test_web_pages.TestWebPage.test_catalog failed
>
>
> Key: IMPALA-10125
> URL: https://issues.apache.org/jira/browse/IMPALA-10125
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Reporter: Yongzhi Chen
>Priority: Major
> Labels: broken-build, flaky
>
> In master-core-data-load, webserver.test_web_pages.TestWebPage.test_catalog
> failed with
> {noformat}
> Stacktrace
> webserver/test_web_pages.py:303: in test_catalog
> self.__test_table_metrics(unique_database, "foo_part", "alter-duration")
> webserver/test_web_pages.py:352: in __test_table_metrics
> "?name=%s.%s" % (db_name, tbl_name), metric,
> ports_to_test=self.CATALOG_TEST_PORT)
> webserver/test_web_pages.py:170: in get_and_check_status
> assert string_to_search in response.text, "URL: {0}
> Str:'{1}'\nResp:{2}".format(
> E AssertionError: URL:
> http://localhost:25020/table_metrics?name=test_catalog_caf8ffd1.foo_part
> Str:'alter-duration'
> E Resp:
> E
> E
> E
> E
> E Apache Impala
> E
> E
> E
> E href="/www/datatables-1.10.18.min.css"/>
> E src="/www/datatables-1.10.18.min.js">
> E rel='stylesheet' media='screen'>
> E
> E
> E @media (min-width: 1300px) {
> E #nav-options {
> E width: 1280px;
> E }
> E }
> E
> E body {
> E font-size: 14px;
> E }
> E
> E pre {
> E padding: 10px;
> E font-size: 12px;
> E border: 1px solid #ccc;
> E }
> E
> E /* Avoid unsightly padding around code element */
> E pre.code {
> E padding: 0;
> E }
> E
> E
> E
> E
> E
> E
> E catalogd
> E
> E role="navigation">
> E
> E
> E href="/">/
> E
> E href="/catalog">/catalog
> E
> E href="/jmx">/jmx
> E
> E href="/log_level">/log_level
> E
> E href="/logs">/logs
> E
> E href="/memz">/memz
> E
> E href="/metrics">/metrics
> E
> E href="/operations">/operations
> E
> E href="/profile_docs">/profile_docs
> E
> E href="/rpcz">/rpcz
> E
> E href="/threadz">/threadz
> E
> E href="/varz">/varz
> E
> E
> E
> E
> E
> E
> E
> E
> E // For Apache Knox compatibility, all urls that are accessed by
> javascript should have
> E // their path wrapped with this.
> E function make_url(path) {
> E var root_link = document.getElementById('root-link');
> E var s = root_link.href.split("?");
> E url = s[0] + path;
> E if (s.length > 1) {
> E if (path.includes("?")) {
> E url += "&"
> E } else {
> E url += "?";
> E }
> E url += s[1];
> E }
> E return url;
> E }
> E
> E
> E
> E Metrics for table test_catalog_caf8ffd1.foo_partare not available
> because the table is currently modified by another operation.
> E
> E
> E
> E
> E
> E
> E assert 'alter-duration' in
[jira] [Resolved] (IMPALA-10125) webserver.test_web_pages.TestWebPage.test_catalog failed
[
https://issues.apache.org/jira/browse/IMPALA-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10125.
Resolution: Fixed
I wasn't able to reproduce. If you see this again, please reopen and attach
more logs.
> webserver.test_web_pages.TestWebPage.test_catalog failed
>
>
> Key: IMPALA-10125
> URL: https://issues.apache.org/jira/browse/IMPALA-10125
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Reporter: Yongzhi Chen
>Assignee: Tim Armstrong
>Priority: Major
> Labels: broken-build, flaky
>
> In master-core-data-load, webserver.test_web_pages.TestWebPage.test_catalog
> failed with
> {noformat}
> Stacktrace
> webserver/test_web_pages.py:303: in test_catalog
> self.__test_table_metrics(unique_database, "foo_part", "alter-duration")
> webserver/test_web_pages.py:352: in __test_table_metrics
> "?name=%s.%s" % (db_name, tbl_name), metric,
> ports_to_test=self.CATALOG_TEST_PORT)
> webserver/test_web_pages.py:170: in get_and_check_status
> assert string_to_search in response.text, "URL: {0}
> Str:'{1}'\nResp:{2}".format(
> E AssertionError: URL:
> http://localhost:25020/table_metrics?name=test_catalog_caf8ffd1.foo_part
> Str:'alter-duration'
> E Resp:
> E
> E
> E
> E
> E Apache Impala
> E
> E
> E
> E href="/www/datatables-1.10.18.min.css"/>
> E src="/www/datatables-1.10.18.min.js">
> E rel='stylesheet' media='screen'>
> E
> E
> E @media (min-width: 1300px) {
> E #nav-options {
> E width: 1280px;
> E }
> E }
> E
> E body {
> E font-size: 14px;
> E }
> E
> E pre {
> E padding: 10px;
> E font-size: 12px;
> E border: 1px solid #ccc;
> E }
> E
> E /* Avoid unsightly padding around code element */
> E pre.code {
> E padding: 0;
> E }
> E
> E
> E
> E
> E
> E
> E catalogd
> E
> E role="navigation">
> E
> E
> E href="/">/
> E
> E href="/catalog">/catalog
> E
> E href="/jmx">/jmx
> E
> E href="/log_level">/log_level
> E
> E href="/logs">/logs
> E
> E href="/memz">/memz
> E
> E href="/metrics">/metrics
> E
> E href="/operations">/operations
> E
> E href="/profile_docs">/profile_docs
> E
> E href="/rpcz">/rpcz
> E
> E href="/threadz">/threadz
> E
> E href="/varz">/varz
> E
> E
> E
> E
> E
> E
> E
> E
> E // For Apache Knox compatibility, all urls that are accessed by
> javascript should have
> E // their path wrapped with this.
> E function make_url(path) {
> E var root_link = document.getElementById('root-link');
> E var s = root_link.href.split("?");
> E url = s[0] + path;
> E if (s.length > 1) {
> E if (path.includes("?")) {
> E url += "&"
> E } else {
> E url += "?";
> E }
> E url += s[1];
> E }
> E return url;
> E }
> E
> E
> E
> E Metrics for table test_catalog_caf8ffd1.foo_partare not available
> because the table is currently modified by another operation.
> E
> E
> E
> E
> E
> E
> E assert 'alter-duration' in
[jira] [Work started] (IMPALA-9050) test_scanners.TestScanRangeLengths.test_scan_ranges is flaky for kudu
[
https://issues.apache.org/jira/browse/IMPALA-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-9050 started by Tim Armstrong.
-
> test_scanners.TestScanRangeLengths.test_scan_ranges is flaky for kudu
> -
>
> Key: IMPALA-9050
> URL: https://issues.apache.org/jira/browse/IMPALA-9050
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.4.0
>Reporter: Yongzhi Chen
>Assignee: Tim Armstrong
>Priority: Major
> Labels: build-failure, flaky
>
> Pre-commit build failed for unrelated change:
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/1405/testReport/junit/query_test.test_scanners/TestScanRangeLengths/test_scan_ranges_max_scan_range_length__512___protocol__beeswax___exec_optionbatch_size___0___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0table_format__kudu_none_/
> query_test.test_scanners.TestScanRangeLengths.test_scan_ranges[max_scan_range_length:
> 512 | protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> kudu/none]
> Error Message
> query_test/test_scanners.py:937: in test_scan_ranges
> self.run_test_case('QueryTest/hdfs-tiny-scan', vector)
> common/impala_test_suite.py:621: in run_test_case result = exec_fn(query,
> user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:556: in __exec_in_impala result =
> self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:893: in __execute_query return
> impalad_client.execute(query, user=user) common/impala_connection.py:205: in
> execute return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:205: in execute result =
> self.fetch_results(query_string, handle) beeswax/impala_beeswax.py:451: in
> fetch_results exec_result = self.__fetch_results(query_handle, max_rows)
> beeswax/impala_beeswax.py:462: in __fetch_results results =
> self.__do_rpc(lambda: self.imp_service.fetch(handle, False, fetch_rows))
> beeswax/impala_beeswax.py:519: in __do_rpc raise
> ImpalaBeeswaxException(self.__build_error_message(b), b) E
> ImpalaBeeswaxException: ImpalaBeeswaxException: EINNER EXCEPTION: 'beeswaxd.ttypes.BeeswaxException'> EMESSAGE: Unable to open Kudu table:
> Network error: failed to read from TLS socket (remote: 172.18.0.1:7051):
> Cannot send after transport endpoint shutdown (error 108)
> Stacktrace
> query_test/test_scanners.py:937: in test_scan_ranges
> self.run_test_case('QueryTest/hdfs-tiny-scan', vector)
> common/impala_test_suite.py:621: in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:556: in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:893: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:205: in execute
> result = self.fetch_results(query_string, handle)
> beeswax/impala_beeswax.py:451: in fetch_results
> exec_result = self.__fetch_results(query_handle, max_rows)
> beeswax/impala_beeswax.py:462: in __fetch_results
> results = self.__do_rpc(lambda: self.imp_service.fetch(handle, False,
> fetch_rows))
> beeswax/impala_beeswax.py:519: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: Unable to open Kudu table: Network error: failed to read from
> TLS socket (remote: 172.18.0.1:7051): Cannot send after transport endpoint
> shutdown (error 108)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-10312) test_shell_interactive.TestImpalaShellInteractive AssertionError: alter query should be closed
[
https://issues.apache.org/jira/browse/IMPALA-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-10312 started by Tim Armstrong.
--
> test_shell_interactive.TestImpalaShellInteractive AssertionError: alter query
> should be closed
> --
>
> Key: IMPALA-10312
> URL: https://issues.apache.org/jira/browse/IMPALA-10312
> Project: IMPALA
> Issue Type: Bug
>Reporter: Kurt Deschler
>Assignee: Tim Armstrong
>Priority: Major
> Labels: broken-build, flaky
>
> shell.test_shell_interactive.TestImpalaShellInteractive.test_ddl_queries_are_closed[table_format_and_file_extension:
> ('textfile', '.txt') | protocol: beeswax] (from pytest)
> Failing for the past 1 build (Since
> [!https://master-02.jenkins.cloudera.com/static/4975bb2b/images/16x16/red.png!
>
> #196|https://master-02.jenkins.cloudera.com/view/Impala/view/Evergreen-asf-master/job/impala-asf-master-core-s3-data-cache/lastCompletedBuild/]
> )
> [Took 22
> sec.|https://master-02.jenkins.cloudera.com/view/Impala/view/Evergreen-asf-master/job/impala-asf-master-core-s3-data-cache/lastCompletedBuild/testReport/shell.test_shell_interactive/TestImpalaShellInteractive/test_ddl_queries_are_closed_table_format_and_file_extensiontextfile_txt_protocol__beeswax_/history]
>
> [!https://master-02.jenkins.cloudera.com/static/4975bb2b/images/16x16/notepad.png!
> add
> description|https://master-02.jenkins.cloudera.com/view/Impala/view/Evergreen-asf-master/job/impala-asf-master-core-s3-data-cache/lastCompletedBuild/testReport/shell.test_shell_interactive/TestImpalaShellInteractive/test_ddl_queries_are_closed_table_format_and_file_extensiontextfile_txt_protocol__beeswax_/editDescription]
> h3. Error Message
> AssertionError: alter query should be closed assert ImpaladService.wait_for_num_in_flight_queries of
> >(0) +
> where > =
> 0x7f49541b2150>.wait_for_num_in_flight_queries
> h3. Stacktrace
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/tests/shell/test_shell_interactive.py:467:
> in test_ddl_queries_are_closed assert
> impalad.wait_for_num_in_flight_queries(0), MSG % 'alter' E AssertionError:
> alter query should be closed E assert ImpaladService.wait_for_num_in_flight_queries of
> >(0) E +
> where > =
> 0x7f49541b2150>.wait_for_num_in_flight_queries
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10312) test_shell_interactive.TestImpalaShellInteractive AssertionError: alter query should be closed
[
https://issues.apache.org/jira/browse/IMPALA-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236556#comment-17236556
]
Tim Armstrong commented on IMPALA-10312:
I don't have logs for this, so I'll try to reproduce it locally and close if I
can't.
> test_shell_interactive.TestImpalaShellInteractive AssertionError: alter query
> should be closed
> --
>
> Key: IMPALA-10312
> URL: https://issues.apache.org/jira/browse/IMPALA-10312
> Project: IMPALA
> Issue Type: Bug
>Reporter: Kurt Deschler
>Assignee: Tim Armstrong
>Priority: Major
> Labels: broken-build, flaky
>
> shell.test_shell_interactive.TestImpalaShellInteractive.test_ddl_queries_are_closed[table_format_and_file_extension:
> ('textfile', '.txt') | protocol: beeswax] (from pytest)
> Failing for the past 1 build (Since
> [!https://master-02.jenkins.cloudera.com/static/4975bb2b/images/16x16/red.png!
>
> #196|https://master-02.jenkins.cloudera.com/view/Impala/view/Evergreen-asf-master/job/impala-asf-master-core-s3-data-cache/lastCompletedBuild/]
> )
> [Took 22
> sec.|https://master-02.jenkins.cloudera.com/view/Impala/view/Evergreen-asf-master/job/impala-asf-master-core-s3-data-cache/lastCompletedBuild/testReport/shell.test_shell_interactive/TestImpalaShellInteractive/test_ddl_queries_are_closed_table_format_and_file_extensiontextfile_txt_protocol__beeswax_/history]
>
> [!https://master-02.jenkins.cloudera.com/static/4975bb2b/images/16x16/notepad.png!
> add
> description|https://master-02.jenkins.cloudera.com/view/Impala/view/Evergreen-asf-master/job/impala-asf-master-core-s3-data-cache/lastCompletedBuild/testReport/shell.test_shell_interactive/TestImpalaShellInteractive/test_ddl_queries_are_closed_table_format_and_file_extensiontextfile_txt_protocol__beeswax_/editDescription]
> h3. Error Message
> AssertionError: alter query should be closed assert ImpaladService.wait_for_num_in_flight_queries of
> >(0) +
> where > =
> 0x7f49541b2150>.wait_for_num_in_flight_queries
> h3. Stacktrace
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/tests/shell/test_shell_interactive.py:467:
> in test_ddl_queries_are_closed assert
> impalad.wait_for_num_in_flight_queries(0), MSG % 'alter' E AssertionError:
> alter query should be closed E assert ImpaladService.wait_for_num_in_flight_queries of
> >(0) E +
> where > =
> 0x7f49541b2150>.wait_for_num_in_flight_queries
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-9049) TestDdlStatements.test_sync_ddl_drop is flaky
[
https://issues.apache.org/jira/browse/IMPALA-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-9049:
-
Assignee: Vihang Karajgaonkar (was: Tim Armstrong)
> TestDdlStatements.test_sync_ddl_drop is flaky
> -
>
> Key: IMPALA-9049
> URL: https://issues.apache.org/jira/browse/IMPALA-9049
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.4.0
>Reporter: Yongzhi Chen
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: broken-build, flaky
>
> It failed precommit tests for unrelated change:
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/1405/testReport/metadata.test_ddl/TestDdlStatements/test_sync_ddl_drop_protocol__beeswax___exec_optionsync_ddl___0___batch_size___0___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0table_format__text_none_unique_database0_/
> Error Message
> test setup failure
> Stacktrace
> conftest.py:319: in cleanup
> {'sync_ddl': sync_ddl})
> common/impala_test_suite.py:790: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:798: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options, user)
> common/impala_test_suite.py:893: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:362: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:356: in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:519: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: AnalysisException: Database does not exist:
> test_sync_ddl_drop_eb3f13e1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-10156) test_unmatched_schema flaky with wrong results
[
https://issues.apache.org/jira/browse/IMPALA-10156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-10156 started by Tim Armstrong.
--
> test_unmatched_schema flaky with wrong results
> --
>
> Key: IMPALA-10156
> URL: https://issues.apache.org/jira/browse/IMPALA-10156
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> Error Message
> query_test/test_scanners.py:244: in test_unmatched_schema
> self.run_test_case('QueryTest/test-unmatched-schema', vector)
> common/impala_test_suite.py:693: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:529: in __verify_results_and_errors
> replace_filenames_with_placeholder) common/test_result_verifier.py:456: in
> verify_raw_results VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results E assert Comparing
> QueryTestResults (expected vs actual): E 1001,'Name1',94611,5000 ==
> 1001,'Name1',94611,5000 E 1002,'Name2',94611,5000 !=
> 1001,'Name1',94611,5000 E 1003,'Name3',94611,5000 !=
> 1002,'Name2',94611,5000 E 1004,'Name4',94611,5000 !=
> 1002,'Name2',94611,5000 E 1005,'Name5',94611,5000 !=
> 1003,'Name3',94611,5000 E 1006,'Name16',94612,15000 !=
> 1003,'Name3',94611,5000 E 1006,'Name16',94612,5000 !=
> 1004,'Name4',94611,5000 E 1006,'Name16',94616,15000 !=
> 1004,'Name4',94611,5000 E 1006,'Name16',94616,5000 !=
> 1005,'Name5',94611,5000 E 1006,'Name6',94616,15000 !=
> 1005,'Name5',94611,5000 E 1006,'Name6',94616,5000 !=
> 1006,'Name16',94612,15000 E 1106,'Name16',94612,15000 !=
> 1006,'Name16',94612,15000 E 1106,'Name16',94612,5000 !=
> 1006,'Name16',94612,5000 E 1106,'Name16',94616,15000 !=
> 1006,'Name16',94612,5000 E 1106,'Name16',94616,5000 !=
> 1006,'Name16',94616,15000 E 1106,'Name6',94612,15000 !=
> 1006,'Name16',94616,15000 E 1106,'Name6',94612,5000 !=
> 1006,'Name16',94616,5000 E 1106,'Name6',94616,15000 !=
> 1006,'Name16',94616,5000 E 1106,'Name6',94616,5000 !=
> 1006,'Name6',94616,15000 E None != 1006,'Name6',94616,15000 E None !=
> 1006,'Name6',94616,5000 E None != 1006,'Name6',94616,5000 E None !=
> 1106,'Name16',94612,15000 E None != 1106,'Name16',94612,15000 E None
> != 1106,'Name16',94612,5000 E None != 1106,'Name16',94612,5000 E None
> != 1106,'Name16',94616,15000 E None != 1106,'Name16',94616,15000 E
> None != 1106,'Name16',94616,5000 E None != 1106,'Name16',94616,5000 E
> None != 1106,'Name6',94612,15000 E None != 1106,'Name6',94612,15000 E
> None != 1106,'Name6',94612,5000 E None != 1106,'Name6',94612,5000 E
> None != 1106,'Name6',94616,15000 E None != 1106,'Name6',94616,15000 E
> None != 1106,'Name6',94616,5000 E None != 1106,'Name6',94616,5000 E
> Number of rows returned (expected vs actual): 19 != 38
> Stacktrace
> query_test/test_scanners.py:244: in test_unmatched_schema
> self.run_test_case('QueryTest/test-unmatched-schema', vector)
> common/impala_test_suite.py:693: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:529: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 1001,'Name1',94611,5000 == 1001,'Name1',94611,5000
> E 1002,'Name2',94611,5000 != 1001,'Name1',94611,5000
> E 1003,'Name3',94611,5000 != 1002,'Name2',94611,5000
> E 1004,'Name4',94611,5000 != 1002,'Name2',94611,5000
> E 1005,'Name5',94611,5000 != 1003,'Name3',94611,5000
> E 1006,'Name16',94612,15000 != 1003,'Name3',94611,5000
> E 1006,'Name16',94612,5000 != 1004,'Name4',94611,5000
> E 1006,'Name16',94616,15000 != 1004,'Name4',94611,5000
> E 1006,'Name16',94616,5000 != 1005,'Name5',94611,5000
> E 1006,'Name6',94616,15000 != 1005,'Name5',94611,5000
> E 1006,'Name6',94616,5000 != 1006,'Name16',94612,15000
> E 1106,'Name16',94612,15000 != 1006,'Name16',94612,15000
> E 1106,'Name16',94612,5000 != 1006,'Name16',94612,5000
> E 1106,'Name16',94616,15000 != 1006,'Name16',94612,5000
> E 1106,'Name16',94616,5000 != 1006,'Name16',94616,15000
> E 1106,'Name6',94612,15000 !=
[jira] [Commented] (IMPALA-10156) test_unmatched_schema flaky with wrong results
[
https://issues.apache.org/jira/browse/IMPALA-10156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236511#comment-17236511
]
Tim Armstrong commented on IMPALA-10156:
It looks like the data is doubled up, so I guess it's more likely a data
loading issue than a query execution issue. This test isn't following best
practices to use unique_database, so I guess I'll try switching to that.
i don't have logs from this job so may be hard to debug further.
> test_unmatched_schema flaky with wrong results
> --
>
> Key: IMPALA-10156
> URL: https://issues.apache.org/jira/browse/IMPALA-10156
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> Error Message
> query_test/test_scanners.py:244: in test_unmatched_schema
> self.run_test_case('QueryTest/test-unmatched-schema', vector)
> common/impala_test_suite.py:693: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:529: in __verify_results_and_errors
> replace_filenames_with_placeholder) common/test_result_verifier.py:456: in
> verify_raw_results VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results E assert Comparing
> QueryTestResults (expected vs actual): E 1001,'Name1',94611,5000 ==
> 1001,'Name1',94611,5000 E 1002,'Name2',94611,5000 !=
> 1001,'Name1',94611,5000 E 1003,'Name3',94611,5000 !=
> 1002,'Name2',94611,5000 E 1004,'Name4',94611,5000 !=
> 1002,'Name2',94611,5000 E 1005,'Name5',94611,5000 !=
> 1003,'Name3',94611,5000 E 1006,'Name16',94612,15000 !=
> 1003,'Name3',94611,5000 E 1006,'Name16',94612,5000 !=
> 1004,'Name4',94611,5000 E 1006,'Name16',94616,15000 !=
> 1004,'Name4',94611,5000 E 1006,'Name16',94616,5000 !=
> 1005,'Name5',94611,5000 E 1006,'Name6',94616,15000 !=
> 1005,'Name5',94611,5000 E 1006,'Name6',94616,5000 !=
> 1006,'Name16',94612,15000 E 1106,'Name16',94612,15000 !=
> 1006,'Name16',94612,15000 E 1106,'Name16',94612,5000 !=
> 1006,'Name16',94612,5000 E 1106,'Name16',94616,15000 !=
> 1006,'Name16',94612,5000 E 1106,'Name16',94616,5000 !=
> 1006,'Name16',94616,15000 E 1106,'Name6',94612,15000 !=
> 1006,'Name16',94616,15000 E 1106,'Name6',94612,5000 !=
> 1006,'Name16',94616,5000 E 1106,'Name6',94616,15000 !=
> 1006,'Name16',94616,5000 E 1106,'Name6',94616,5000 !=
> 1006,'Name6',94616,15000 E None != 1006,'Name6',94616,15000 E None !=
> 1006,'Name6',94616,5000 E None != 1006,'Name6',94616,5000 E None !=
> 1106,'Name16',94612,15000 E None != 1106,'Name16',94612,15000 E None
> != 1106,'Name16',94612,5000 E None != 1106,'Name16',94612,5000 E None
> != 1106,'Name16',94616,15000 E None != 1106,'Name16',94616,15000 E
> None != 1106,'Name16',94616,5000 E None != 1106,'Name16',94616,5000 E
> None != 1106,'Name6',94612,15000 E None != 1106,'Name6',94612,15000 E
> None != 1106,'Name6',94612,5000 E None != 1106,'Name6',94612,5000 E
> None != 1106,'Name6',94616,15000 E None != 1106,'Name6',94616,15000 E
> None != 1106,'Name6',94616,5000 E None != 1106,'Name6',94616,5000 E
> Number of rows returned (expected vs actual): 19 != 38
> Stacktrace
> query_test/test_scanners.py:244: in test_unmatched_schema
> self.run_test_case('QueryTest/test-unmatched-schema', vector)
> common/impala_test_suite.py:693: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:529: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 1001,'Name1',94611,5000 == 1001,'Name1',94611,5000
> E 1002,'Name2',94611,5000 != 1001,'Name1',94611,5000
> E 1003,'Name3',94611,5000 != 1002,'Name2',94611,5000
> E 1004,'Name4',94611,5000 != 1002,'Name2',94611,5000
> E 1005,'Name5',94611,5000 != 1003,'Name3',94611,5000
> E 1006,'Name16',94612,15000 != 1003,'Name3',94611,5000
> E 1006,'Name16',94612,5000 != 1004,'Name4',94611,5000
> E 1006,'Name16',94616,15000 != 1004,'Name4',94611,5000
> E 1006,'Name16',94616,5000 != 1005,'Name5',94611,5000
> E 1006,'Name6',94616,15000 != 1005,'Name5',94611,5000
> E 1006,'Name6',94616,5000 !=
[jira] [Updated] (IMPALA-9991) TestShellClient.test_fetch_size_result_spooling is flaky
[
https://issues.apache.org/jira/browse/IMPALA-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-9991:
--
Issue Type: Bug (was: Test)
> TestShellClient.test_fetch_size_result_spooling is flaky
>
>
> Key: IMPALA-9991
> URL: https://issues.apache.org/jira/browse/IMPALA-9991
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
>
> shell.test_shell_client.TestShellClient.test_fetch_size_result_spooling[table_format_and_file_extension:
> ('parquet', '.parq') | protocol: hs2] (from pytest)
> h3. Error Message
> shell/test_shell_client.py:70: in test_fetch_size_result_spooling
> self.__fetch_rows(client.fetch(handle), num_rows / fetch_size, num_rows)
> shell/test_shell_client.py:80: in __fetch_rows for fetch_batch in
> fetch_batches: ../shell/impala_client.py:787: in fetch yield
> self._transpose(col_value_converters, resp.results.columns) E AttributeError:
> 'NoneType' object has no attribute 'columns'
> h3. Stacktrace
> shell/test_shell_client.py:70: in test_fetch_size_result_spooling
> self.__fetch_rows(client.fetch(handle), num_rows / fetch_size, num_rows)
> shell/test_shell_client.py:80: in __fetch_rows for fetch_batch in
> fetch_batches: ../shell/impala_client.py:787: in fetch yield
> self._transpose(col_value_converters, resp.results.columns) E AttributeError:
> 'NoneType' object has no attribute 'columns'
> h3. Standard Error
> Opened TCP connection to localhost:21050
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9991) TestShellClient.test_fetch_size_result_spooling is flaky
[
https://issues.apache.org/jira/browse/IMPALA-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-9991:
--
Priority: Critical (was: Major)
> TestShellClient.test_fetch_size_result_spooling is flaky
>
>
> Key: IMPALA-9991
> URL: https://issues.apache.org/jira/browse/IMPALA-9991
> Project: IMPALA
> Issue Type: Test
>Reporter: Sahil Takiar
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
>
> shell.test_shell_client.TestShellClient.test_fetch_size_result_spooling[table_format_and_file_extension:
> ('parquet', '.parq') | protocol: hs2] (from pytest)
> h3. Error Message
> shell/test_shell_client.py:70: in test_fetch_size_result_spooling
> self.__fetch_rows(client.fetch(handle), num_rows / fetch_size, num_rows)
> shell/test_shell_client.py:80: in __fetch_rows for fetch_batch in
> fetch_batches: ../shell/impala_client.py:787: in fetch yield
> self._transpose(col_value_converters, resp.results.columns) E AttributeError:
> 'NoneType' object has no attribute 'columns'
> h3. Stacktrace
> shell/test_shell_client.py:70: in test_fetch_size_result_spooling
> self.__fetch_rows(client.fetch(handle), num_rows / fetch_size, num_rows)
> shell/test_shell_client.py:80: in __fetch_rows for fetch_batch in
> fetch_batches: ../shell/impala_client.py:787: in fetch yield
> self._transpose(col_value_converters, resp.results.columns) E AttributeError:
> 'NoneType' object has no attribute 'columns'
> h3. Standard Error
> Opened TCP connection to localhost:21050
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-9991) TestShellClient.test_fetch_size_result_spooling is flaky
[
https://issues.apache.org/jira/browse/IMPALA-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-9991:
-
Assignee: Riza Suminto (was: Tim Armstrong)
> TestShellClient.test_fetch_size_result_spooling is flaky
>
>
> Key: IMPALA-9991
> URL: https://issues.apache.org/jira/browse/IMPALA-9991
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Riza Suminto
>Priority: Critical
> Labels: broken-build, flaky
>
> shell.test_shell_client.TestShellClient.test_fetch_size_result_spooling[table_format_and_file_extension:
> ('parquet', '.parq') | protocol: hs2] (from pytest)
> h3. Error Message
> shell/test_shell_client.py:70: in test_fetch_size_result_spooling
> self.__fetch_rows(client.fetch(handle), num_rows / fetch_size, num_rows)
> shell/test_shell_client.py:80: in __fetch_rows for fetch_batch in
> fetch_batches: ../shell/impala_client.py:787: in fetch yield
> self._transpose(col_value_converters, resp.results.columns) E AttributeError:
> 'NoneType' object has no attribute 'columns'
> h3. Stacktrace
> shell/test_shell_client.py:70: in test_fetch_size_result_spooling
> self.__fetch_rows(client.fetch(handle), num_rows / fetch_size, num_rows)
> shell/test_shell_client.py:80: in __fetch_rows for fetch_batch in
> fetch_batches: ../shell/impala_client.py:787: in fetch yield
> self._transpose(col_value_converters, resp.results.columns) E AttributeError:
> 'NoneType' object has no attribute 'columns'
> h3. Standard Error
> Opened TCP connection to localhost:21050
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9991) TestShellClient.test_fetch_size_result_spooling is flaky
[
https://issues.apache.org/jira/browse/IMPALA-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236506#comment-17236506
]
Tim Armstrong commented on IMPALA-9991:
---
[~rizaon] while you're looking at result spooling, can you see if this test is
flaky? Don't spend too much time if it's not reproducible.
> TestShellClient.test_fetch_size_result_spooling is flaky
>
>
> Key: IMPALA-9991
> URL: https://issues.apache.org/jira/browse/IMPALA-9991
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Riza Suminto
>Priority: Critical
> Labels: broken-build, flaky
>
> shell.test_shell_client.TestShellClient.test_fetch_size_result_spooling[table_format_and_file_extension:
> ('parquet', '.parq') | protocol: hs2] (from pytest)
> h3. Error Message
> shell/test_shell_client.py:70: in test_fetch_size_result_spooling
> self.__fetch_rows(client.fetch(handle), num_rows / fetch_size, num_rows)
> shell/test_shell_client.py:80: in __fetch_rows for fetch_batch in
> fetch_batches: ../shell/impala_client.py:787: in fetch yield
> self._transpose(col_value_converters, resp.results.columns) E AttributeError:
> 'NoneType' object has no attribute 'columns'
> h3. Stacktrace
> shell/test_shell_client.py:70: in test_fetch_size_result_spooling
> self.__fetch_rows(client.fetch(handle), num_rows / fetch_size, num_rows)
> shell/test_shell_client.py:80: in __fetch_rows for fetch_batch in
> fetch_batches: ../shell/impala_client.py:787: in fetch yield
> self._transpose(col_value_converters, resp.results.columns) E AttributeError:
> 'NoneType' object has no attribute 'columns'
> h3. Standard Error
> Opened TCP connection to localhost:21050
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10339) Apparent hang or crash in TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action
[
https://issues.apache.org/jira/browse/IMPALA-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236505#comment-17236505
]
Tim Armstrong commented on IMPALA-10339:
I looped this overnight on master and wasn't able to hit the DCHECK or a hang.
> Apparent hang or crash in
> TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action
> ---
>
> Key: IMPALA-10339
> URL: https://issues.apache.org/jira/browse/IMPALA-10339
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build, flaky, hang
>
> Release build with this commit as the tip:
> {noformat}
> commit 9400e9b17b13f613defb6d7b9deb471256b1d95c (CDH/cdpd-master-staging)
> Author: wzhou-code
> Date: Thu Oct 29 22:32:03 2020 -0700
> IMPALA-10305: Sync Kudu's FIPS compliant changes
>
> {noformat}
> {noformat}
> Regression
> query_test.test_spilling.TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action[protocol:
> beeswax | exec_option: {'mt_dop': 0, 'default_spillable_buffer_size': '64k'}
> | table_format: parquet/none] (from pytest)
> Failing for the past 1 build (Since Failed#100 )
> Took 1 hr 59 min.
> add description
> Error Message
> query_test/test_spilling.py:134: in test_spilling_no_debug_action
> self.run_test_case('QueryTest/spilling-no-debug-action', vector)
> common/impala_test_suite.py:668: in run_test_case
> self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> common/impala_test_suite.py:485: in __verify_exceptions (expected_str,
> actual_str) E AssertionError: Unexpected exception string. Expected:
> row_regex:.*Cannot perform hash join at node with id .*. Repartitioning did
> not reduce the size of a spilled partition.* E Not found in actual: Timeout
> >7200s
> Stacktrace
> query_test/test_spilling.py:134: in test_spilling_no_debug_action
> self.run_test_case('QueryTest/spilling-no-debug-action', vector)
> common/impala_test_suite.py:668: in run_test_case
> self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> common/impala_test_suite.py:485: in __verify_exceptions
> (expected_str, actual_str)
> E AssertionError: Unexpected exception string. Expected: row_regex:.*Cannot
> perform hash join at node with id .*. Repartitioning did not reduce the size
> of a spilled partition.*
> E Not found in actual: Timeout >7200s
> Standard Error
> SET
> client_identifier=query_test/test_spilling.py::TestSpillingNoDebugActionDimensions::()::test_spilling_no_debug_action[protocol:beeswax|exec_option:{'mt_dop':0;'default_spillable_buffer_size':'64k'}|table_format:parquet/none];
> -- executing against localhost:21000
> use tpch_parquet;
> -- 2020-11-11 23:12:04,319 INFO MainThread: Started query
> c740c1c66d9679a9:6a40f161
> SET
> client_identifier=query_test/test_spilling.py::TestSpillingNoDebugActionDimensions::()::test_spilling_no_debug_action[protocol:beeswax|exec_option:{'mt_dop':0;'default_spillable_buffer_size':'64k'}|table_format:parquet/none];
> SET mt_dop=0;
> SET default_spillable_buffer_size=64k;
> -- 2020-11-11 23:12:04,320 INFO MainThread: Loading query test file:
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/testdata/workloads/functional-query/queries/QueryTest/spilling-no-debug-action.test
> -- 2020-11-11 23:12:04,323 INFO MainThread: Starting new HTTP connection
> (1): localhost
> -- executing against localhost:21000
> set debug_action="-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0";
> -- 2020-11-11 23:12:04,377 INFO MainThread: Started query
> c044afcf5ae44df9:a2e2e7c6
> -- executing against localhost:21000
> select straight_join count(*)
> from
> lineitem a, lineitem b
> where
> a.l_partkey = 1 and
> a.l_orderkey = b.l_orderkey;
> -- 2020-11-11 23:12:04,385 INFO MainThread: Started query
> 314c019cd252f322:2411bc76
> -- executing against localhost:21000
> SET DEBUG_ACTION="";
> -- 2020-11-11 23:12:05,199 INFO MainThread: Started query
> 80424e68922c30f9:b2144dff
> -- executing against localhost:21000
> set debug_action="-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0";
> -- 2020-11-11 23:12:05,207 INFO MainThread: Started query
> 2a4c1f4b26ea52da:4339f3ff
> -- executing against localhost:21000
> select straight_join count(*)
> from
> lineitem a
> where
> a.l_partkey not in (select l_partkey from lineitem where l_partkey > 10)
> and a.l_partkey < 1000;
> -- 2020-11-11 23:12:05,215 INFO MainThread: Started query
> f845afd00a569446:79c5054a
> -- executing against localhost:21000
> SET DEBUG_ACTION="";
> -- 2020-11-11 23:12:07,507
[jira] [Commented] (IMPALA-9884) TestAdmissionControllerStress.test_mem_limit failing occasionally
[
https://issues.apache.org/jira/browse/IMPALA-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236504#comment-17236504
]
Tim Armstrong commented on IMPALA-9884:
---
FWIW this seems to mainly be a problem on release builds.
> TestAdmissionControllerStress.test_mem_limit failing occasionally
> -
>
> Key: IMPALA-9884
> URL: https://issues.apache.org/jira/browse/IMPALA-9884
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build, flaky
> Attachments: impalad-executors.tar.gz,
> impalad.impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com.jenkins.log.INFO.20201017-06.23933.gz
>
>
> Recently, I saw this test failing with the exception trace below.
> {noformat}
> custom_cluster/test_admission_controller.py:1782: in test_mem_limit
> {'request_pool': self.pool_name, 'mem_limit': query_mem_limit})
> custom_cluster/test_admission_controller.py:1638: in run_admission_test
> assert metric_deltas['dequeued'] == 0,\
> E AssertionError: Queued queries should not run until others are made to
> finish
> E assert 1 == 0
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10194) ASAN use-after-free in hdfs-util-test
[
https://issues.apache.org/jira/browse/IMPALA-10194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10194.
Resolution: Duplicate
> ASAN use-after-free in hdfs-util-test
> -
>
> Key: IMPALA-10194
> URL: https://issues.apache.org/jira/browse/IMPALA-10194
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Laszlo Gaal
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build, flaky
>
> This is the ASAN report section from LastTest.log:
> {code}
> 90/124 Testing: hdfs-util-test
> 90/124 Test: hdfs-util-test
> Command:
> "/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/bin/run-jvm-binary.sh"
>
> "/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug//util/hdfs-util-test"
>
> "-log_dir=/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/be_tests"
> Directory:
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util
> "hdfs-util-test" start time: Sep 25 22:18 PDT
> Output:
> --
> seed = 1601097520
> Note: Google Test filter = HdfsUtilTest.*
> [==] Running 2 tests from 1 test case.
> [--] Global test environment set-up.
> [--] 2 tests from HdfsUtilTest
> [ RUN ] HdfsUtilTest.CheckFilesystemsMatch
> 20/09/25 22:18:40 INFO util.JvmPauseMonitor: Starting JVM pause monitor
> [ OK ] HdfsUtilTest.CheckFilesystemsMatch (941 ms)
> [ RUN ] HdfsUtilTest.CheckGetBaseName
> [ OK ] HdfsUtilTest.CheckGetBaseName (0 ms)
> [--] 2 tests from HdfsUtilTest (941 ms total)
> [--] Global test environment tear-down
> [==] 2 tests from 1 test case ran. (941 ms total)
> [ PASSED ] 2 tests.
> YOU HAVE 1 DISABLED TEST
> =
> ==17289==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x6070e440 at pc 0x01e3854b bp 0x7effcdf609e0 sp 0x7effcdf60190
> READ of size 7 at 0x6070e440 thread T4
>
> Test time = 2.13 sec
> --
> Test Passed.
> "hdfs-util-test" end time: Sep 25 22:18 PDT
> "hdfs-util-test" time elapsed: 00:00:02
> --
> {code}
> The following two commits have been added to master since the last green ASAN
> run:
> {code}
> IMPALA-10183: Fix hitting DCHECK when cancelling a query with result spooling
> IMPALA-10170: Data race on Webserver::UrlHandler::is_on_nav_bar_
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9865) Utility to pretty-print thrift profiles at various levels
[ https://issues.apache.org/jira/browse/IMPALA-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236454#comment-17236454 ] Tim Armstrong commented on IMPALA-9865: --- The problem we have with that is that some tools do depend on the detailed counters for certain features. E.g. cloudera manager aggregates CPU, memory, BytesRead, etc from the entire profile to report those. So we want to make sure that the information is available in some form. > Utility to pretty-print thrift profiles at various levels > - > > Key: IMPALA-9865 > URL: https://issues.apache.org/jira/browse/IMPALA-9865 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Attachments: image-2020-11-20-09-25-08-082.png > > > The prototyping work in IMPALA-9382 revealed some hard trade-offs between > having a full-fidelity text profile and readability. > We want to have a text profile with less information by default so that it is > more readable, and rely on the thrift profile for more detailed debugging. > This would be easier if we provided a utility that can pretty-print a thrift > profile at different levels of detail. > This JIRA is to reduce the default level of pretty-printing for aggregated > profiles, but provide a utility that can dump both the basic and full > versions. My thought is to start off with the same 4 levels as explain, but > maybe only implement 2 levels to start off with - basic and extended. > The utility should be able to handle the same cases as > bin/parse-thrift-profile.py (profile log and single profile in a file) and > maybe print only a specified query from a profile log. We can use the > DeserializeFromArchiveString() method that was removed in IMPALA-9381, then > pretty-print the deserialised profile. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10347) Explore approaches to optimizing queries that will likely be short-circuited by limits
Tim Armstrong created IMPALA-10347: -- Summary: Explore approaches to optimizing queries that will likely be short-circuited by limits Key: IMPALA-10347 URL: https://issues.apache.org/jira/browse/IMPALA-10347 Project: IMPALA Issue Type: Improvement Components: Distributed Exec Reporter: Tim Armstrong Based on discussion with [~amansinha], there are opportunities beyond IMPALA-10314 to optimize queries where there is a limit and the query is *unlikely* to scan many files. The problem is that we do all the work to generate scan ranges and schedule them upfront, which adds a lot of overhead if only a small number of files actually need to be processed. A couple of ideas we had: * Parallelize and/or otherwise optimize the scan range generation * Speculatively execute the query on a subset of files and then cancel and retry if we hit the limit * Incrementally generate scan ranges and assign them to executors so that scan range generation and execution can be overlapped. This is the most general solution but also has a lot of knock-on implications for other subsystems, like cardinality/memory estimation, scheduling, query execution, query coordination, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9437) Cannot get Ozone file block size after opening the file
[
https://issues.apache.org/jira/browse/IMPALA-9437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236324#comment-17236324
]
Tim Armstrong commented on IMPALA-9437:
---
[~adoroszlai][~arp] here's the relevant code:
https://github.com/apache/impala/blob/981ef104654e187ae18b25f31df2fd324e687643/be/src/exec/hdfs-table-sink.cc#L402
Here's the logic for parquet default_block_size:
https://github.com/apache/impala/blob/1bd27a3ea87622edc869d3ff72ce9b0881451c95/be/src/exec/parquet/hdfs-parquet-table-writer.cc#L1211
And for text:
https://github.com/apache/impala/blob/8b8a49e617818e9bcf99b784b63587c95cebd622/be/src/exec/hdfs-text-table-writer.cc#L63
And HDFS_BLOCK_SIZE:
https://github.com/apache/impala/blob/981ef104654e187ae18b25f31df2fd324e687643/be/src/exec/parquet/hdfs-parquet-table-writer.h#L94,
which is 256MB, which is the default ozone block size.
The block size is used for sizing parquet files, so that row groups line up to
block boundaries. I think it's a little weird if we get the wrong size, e.g.
the ozone block size is 128MB and the parquet writer thinks it's 256mb - you
get a row group split across blocks, which will make parallelism suboptimal.
I think if we replaced the reference to HDFS_BLOCK_SIZE with a call to a
function that returned the actual configured default for the path being written
to, that would be a generally good improvement.
It does depend a bit on whether people use non-default Ozone block sizes with
any frequency.
> Cannot get Ozone file block size after opening the file
> ---
>
> Key: IMPALA-9437
> URL: https://issues.apache.org/jira/browse/IMPALA-9437
> Project: IMPALA
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Priority: Major
>
> When creating a tmp file on HDFS, {{HdfsTableSink::CreateNewTmpFile}} first
> opens the file, and then stats the file ({{hdfsGetPathInfo}}) before actually
> writing any data to the file or closing the file. HDFS seems to allow this
> behavior. However, Ozone, S3A, and ABFS do not. Impala does this for HDFS in
> order to get the block size of the opened file. According to
> {{HdfsTableSink}} it is possible for HDFS to create a block size with a
> different one than requested by Impala. So in order to track the correct
> block size for a file, the file needs to be stat'ed after opening it. For S3A
> and ABFS this isn't a big deal, because they aren't block based filesystem,
> but Ozone is. So we should investigate the impact of not having this
> capability and consider adding it to the Ozone client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9121) heap-use-after-free in HdfsUtilTest CheckFilesystemsMatch
[
https://issues.apache.org/jira/browse/IMPALA-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-9121.
---
Resolution: Fixed
Closing for now since I won't take any action unless it reoccurs
> heap-use-after-free in HdfsUtilTest CheckFilesystemsMatch
> -
>
> Key: IMPALA-9121
> URL: https://issues.apache.org/jira/browse/IMPALA-9121
> Project: IMPALA
> Issue Type: Bug
>Reporter: Bikramjeet Vig
>Assignee: Tim Armstrong
>Priority: Major
> Labels: asan, broken-build, flaky-test
>
> {noformat}
> --
> 85/113 Testing: hdfs-util-test
> 85/113 Test: hdfs-util-test
> Command:
> "/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/util/hdfs-util-test"
> "-log_dir=/data/jenkins/workspace
> /impala-asf-master-core-asan/repos/Impala/logs/be_tests"
> Directory:
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util
> "hdfs-util-test" start time: Nov 01 22:21 PDT
> Output:
> --
> seed = 1572672103
> Note: Google Test filter = HdfsUtilTest.*
> [==] Running 2 tests from 1 test case.
> [--] Global test environment set-up.
> [--] 2 tests from HdfsUtilTest
> [ RUN ] HdfsUtilTest.CheckFilesystemsMatch
> 19/11/01 22:21:43 INFO util.JvmPauseMonitor: Starting JVM pause monitor
> [ OK ] HdfsUtilTest.CheckFilesystemsMatch (1012 ms)
> [ RUN ] HdfsUtilTest.CheckGetBaseName
> [ OK ] HdfsUtilTest.CheckGetBaseName (0 ms)
> [--] 2 tests from HdfsUtilTest (1012 ms total)
> [--] Global test environment tear-down
> [==] 2 tests from 1 test case ran. (1013 ms total)
> [ PASSED ] 2 tests.
> YOU HAVE 1 DISABLED TEST
> =
> ==87116==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x60362878 at pc 0x01b67bac bp 0x7fc96cf57ae0 sp 0x7fc96cf57290
> READ of size 4 at 0x60362878 thread T4
>
> Test time = 2.64 sec
> --
> Test Passed.
> "hdfs-util-test" end time: Nov 01 22:21 PDT
> "hdfs-util-test" time elapsed: 00:00:02
> --
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9865) Utility to pretty-print thrift profiles at various levels
[ https://issues.apache.org/jira/browse/IMPALA-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235848#comment-17235848 ] Tim Armstrong commented on IMPALA-9865: --- [~fifteencai] this is great feedback. I'm working on a new profile approach that does something like this, except it retains all the information in a denser format. I know the Cloudera Manager team has also been doing some work to handle large profiles better by not keeping them in memory for as long. > Utility to pretty-print thrift profiles at various levels > - > > Key: IMPALA-9865 > URL: https://issues.apache.org/jira/browse/IMPALA-9865 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Attachments: image-2020-11-20-09-25-08-082.png > > > The prototyping work in IMPALA-9382 revealed some hard trade-offs between > having a full-fidelity text profile and readability. > We want to have a text profile with less information by default so that it is > more readable, and rely on the thrift profile for more detailed debugging. > This would be easier if we provided a utility that can pretty-print a thrift > profile at different levels of detail. > This JIRA is to reduce the default level of pretty-printing for aggregated > profiles, but provide a utility that can dump both the basic and full > versions. My thought is to start off with the same 4 levels as explain, but > maybe only implement 2 levels to start off with - basic and extended. > The utility should be able to handle the same cases as > bin/parse-thrift-profile.py (profile log and single profile in a file) and > maybe print only a specified query from a profile log. We can use the > DeserializeFromArchiveString() method that was removed in IMPALA-9381, then > pretty-print the deserialised profile. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10344) Error loading functional_orc_def.alltypesaggmultifiles in hive
[
https://issues.apache.org/jira/browse/IMPALA-10344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-10344:
---
Summary: Error loading functional_orc_def.alltypesaggmultifiles in hive
(was: Error loading functional_orc_def.al ltypesaggmultifiles in hive)
> Error loading functional_orc_def.alltypesaggmultifiles in hive
> --
>
> Key: IMPALA-10344
> URL: https://issues.apache.org/jira/browse/IMPALA-10344
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 4.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
> Attachments: hive-metastore.out, hive-server2.out,
> load-functional-query-exhaustive-hive-generated-orc-def-block.sql.log
>
>
> Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
> Exception when loading 11 partitions in table alltypesaggmultifiles
> {noformat}
> 0: jdbc:hive2://localhost:11050/default> insert into table
> functional_orc_def.al
> ltypesaggmultifiles partition (year, month, day) SELECT id, bool_col,
> tinyint_co
> l, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col,
> st
> ring_col, timestamp_col, year, month, day FROM
> functional.alltypesaggmultifiles
> where id % 4 = 1;
> going to print operations logs
> printed operations logs
> going to print operations logs
> INFO : Compiling
> command(queryId=jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9):
> insert into table functional_orc_def.alltypesaggmultifiles partition (year,
> month, day) SELECT id, bool_col, tinyint_col, smallint_col, int_col,
> bigint_col, float_col, double_col, date_string_col, string_col,
> timestamp_col, year, month, day FROM functional.alltypesaggmultifiles where
> id % 4 = 1
> INFO : No Stats for functional@alltypesaggmultifiles, Columns: float_col,
> int_col, string_col, bool_col, date_string_col, smallint_col, timestamp_col,
> tinyint_col, bigint_col, id, double_col
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:id,
> type:int, comment:null), FieldSchema(name:bool_col, type:boolean,
> comment:null), FieldSchema(name:tinyint_col, type:tinyint, comment:null),
> FieldSchema(name:smallint_col, type:smallint, comment:null),
> FieldSchema(name:int_col, type:int, comment:null),
> FieldSchema(name:bigint_col, type:bigint, comment:null),
> FieldSchema(name:float_col, type:float, comment:null),
> FieldSchema(name:double_col, type:double, comment:null),
> FieldSchema(name:date_string_col, type:string, comment:null),
> FieldSchema(name:string_col, type:string, comment:null),
> FieldSchema(name:timestamp_col, type:timestamp, comment:null),
> FieldSchema(name:year, type:int, comment:null), FieldSchema(name:month,
> type:int, comment:null), FieldSchema(name:day, type:int, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9);
> Time taken: 0.036 seconds
> INFO : Executing
> command(queryId=jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9):
> insert into table functional_orc_def.alltypesaggmultifiles partition (year,
> month, day) SELECT id, bool_col, tinyint_col, smallint_col, int_col,
> bigint_col, float_col, double_col, date_string_col, string_col,
> timestamp_col, year, month, day FROM functional.alltypesaggmultifiles where
> id % 4 = 1
> INFO : Query ID = jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9
> INFO : Total jobs = 1
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-1:MAPRED] in serial mode
> INFO : Subscribed to counters: [] for queryId:
> jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9
> INFO : Session is already open
> INFO : Dag name: insert into table functional_orc_def.all...1 (Stage-1)
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1605304119768_0012)
> printed operations logs
> [2K--
> [2K[36;1mVERTICES MODESTATUS TOTAL COMPLETED
> RUNNING PENDING FAILED KILLED
> [22;0m[2K--
> [2KMap 1 .. container SUCCEEDED 42 420
> 0 0 0
> Reducer 2container RUNNING 1 010
> 0 0
> [2K--
> [2K[31;1mVERTICES: 01/02 [=>>-] 97% ELAPSED
> TIME: 1.46
[jira] [Created] (IMPALA-10344) Error loading functional_orc_def.al ltypesaggmultifiles in hive
Tim Armstrong created IMPALA-10344:
--
Summary: Error loading functional_orc_def.al ltypesaggmultifiles
in hive
Key: IMPALA-10344
URL: https://issues.apache.org/jira/browse/IMPALA-10344
Project: IMPALA
Issue Type: Improvement
Components: Infrastructure
Affects Versions: Impala 4.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong
Attachments: hive-metastore.out, hive-server2.out,
load-functional-query-exhaustive-hive-generated-orc-def-block.sql.log
Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
Exception when loading 11 partitions in table alltypesaggmultifiles
{noformat}
0: jdbc:hive2://localhost:11050/default> insert into table
functional_orc_def.al
ltypesaggmultifiles partition (year, month, day) SELECT id, bool_col,
tinyint_co
l, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col,
st
ring_col, timestamp_col, year, month, day FROM functional.alltypesaggmultifiles
where id % 4 = 1;
going to print operations logs
printed operations logs
going to print operations logs
INFO : Compiling
command(queryId=jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9):
insert into table functional_orc_def.alltypesaggmultifiles partition (year,
month, day) SELECT id, bool_col, tinyint_col, smallint_col, int_col,
bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col,
year, month, day FROM functional.alltypesaggmultifiles where id % 4 = 1
INFO : No Stats for functional@alltypesaggmultifiles, Columns: float_col,
int_col, string_col, bool_col, date_string_col, smallint_col, timestamp_col,
tinyint_col, bigint_col, id, double_col
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:id,
type:int, comment:null), FieldSchema(name:bool_col, type:boolean,
comment:null), FieldSchema(name:tinyint_col, type:tinyint, comment:null),
FieldSchema(name:smallint_col, type:smallint, comment:null),
FieldSchema(name:int_col, type:int, comment:null), FieldSchema(name:bigint_col,
type:bigint, comment:null), FieldSchema(name:float_col, type:float,
comment:null), FieldSchema(name:double_col, type:double, comment:null),
FieldSchema(name:date_string_col, type:string, comment:null),
FieldSchema(name:string_col, type:string, comment:null),
FieldSchema(name:timestamp_col, type:timestamp, comment:null),
FieldSchema(name:year, type:int, comment:null), FieldSchema(name:month,
type:int, comment:null), FieldSchema(name:day, type:int, comment:null)],
properties:null)
INFO : Completed compiling
command(queryId=jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9);
Time taken: 0.036 seconds
INFO : Executing
command(queryId=jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9):
insert into table functional_orc_def.alltypesaggmultifiles partition (year,
month, day) SELECT id, bool_col, tinyint_col, smallint_col, int_col,
bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col,
year, month, day FROM functional.alltypesaggmultifiles where id % 4 = 1
INFO : Query ID = jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Subscribed to counters: [] for queryId:
jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9
INFO : Session is already open
INFO : Dag name: insert into table functional_orc_def.all...1 (Stage-1)
INFO : Status: Running (Executing on YARN cluster with App id
application_1605304119768_0012)
printed operations logs
[2K--
[2K[36;1mVERTICES MODESTATUS TOTAL COMPLETED RUNNING
PENDING FAILED KILLED
[22;0m[2K--
[2KMap 1 .. container SUCCEEDED 42 420
0 0 0
Reducer 2container RUNNING 1 010
0 0
[2K--
[2K[31;1mVERTICES: 01/02 [=>>-] 97% ELAPSED TIME:
1.46 s
[22;0m[2K--
[8A[2K--
[2K[36;1mVERTICES MODESTATUS TOTAL COMPLETED RUNNING
PENDING FAILED KILLED
[22;0m[2K--
[2KMap 1 .. container SUCCEEDED 42 420
0 0 0
Reducer 2container
[jira] [Updated] (IMPALA-10339) Apparent hang or crash in TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action
[
https://issues.apache.org/jira/browse/IMPALA-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-10339:
---
Summary: Apparent hang or crash in
TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action (was:
Apparent hang in
TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action)
> Apparent hang or crash in
> TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action
> ---
>
> Key: IMPALA-10339
> URL: https://issues.apache.org/jira/browse/IMPALA-10339
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build, flaky, hang
>
> Release build with this commit as the tip:
> {noformat}
> commit 9400e9b17b13f613defb6d7b9deb471256b1d95c (CDH/cdpd-master-staging)
> Author: wzhou-code
> Date: Thu Oct 29 22:32:03 2020 -0700
> IMPALA-10305: Sync Kudu's FIPS compliant changes
>
> {noformat}
> {noformat}
> Regression
> query_test.test_spilling.TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action[protocol:
> beeswax | exec_option: {'mt_dop': 0, 'default_spillable_buffer_size': '64k'}
> | table_format: parquet/none] (from pytest)
> Failing for the past 1 build (Since Failed#100 )
> Took 1 hr 59 min.
> add description
> Error Message
> query_test/test_spilling.py:134: in test_spilling_no_debug_action
> self.run_test_case('QueryTest/spilling-no-debug-action', vector)
> common/impala_test_suite.py:668: in run_test_case
> self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> common/impala_test_suite.py:485: in __verify_exceptions (expected_str,
> actual_str) E AssertionError: Unexpected exception string. Expected:
> row_regex:.*Cannot perform hash join at node with id .*. Repartitioning did
> not reduce the size of a spilled partition.* E Not found in actual: Timeout
> >7200s
> Stacktrace
> query_test/test_spilling.py:134: in test_spilling_no_debug_action
> self.run_test_case('QueryTest/spilling-no-debug-action', vector)
> common/impala_test_suite.py:668: in run_test_case
> self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> common/impala_test_suite.py:485: in __verify_exceptions
> (expected_str, actual_str)
> E AssertionError: Unexpected exception string. Expected: row_regex:.*Cannot
> perform hash join at node with id .*. Repartitioning did not reduce the size
> of a spilled partition.*
> E Not found in actual: Timeout >7200s
> Standard Error
> SET
> client_identifier=query_test/test_spilling.py::TestSpillingNoDebugActionDimensions::()::test_spilling_no_debug_action[protocol:beeswax|exec_option:{'mt_dop':0;'default_spillable_buffer_size':'64k'}|table_format:parquet/none];
> -- executing against localhost:21000
> use tpch_parquet;
> -- 2020-11-11 23:12:04,319 INFO MainThread: Started query
> c740c1c66d9679a9:6a40f161
> SET
> client_identifier=query_test/test_spilling.py::TestSpillingNoDebugActionDimensions::()::test_spilling_no_debug_action[protocol:beeswax|exec_option:{'mt_dop':0;'default_spillable_buffer_size':'64k'}|table_format:parquet/none];
> SET mt_dop=0;
> SET default_spillable_buffer_size=64k;
> -- 2020-11-11 23:12:04,320 INFO MainThread: Loading query test file:
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/testdata/workloads/functional-query/queries/QueryTest/spilling-no-debug-action.test
> -- 2020-11-11 23:12:04,323 INFO MainThread: Starting new HTTP connection
> (1): localhost
> -- executing against localhost:21000
> set debug_action="-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0";
> -- 2020-11-11 23:12:04,377 INFO MainThread: Started query
> c044afcf5ae44df9:a2e2e7c6
> -- executing against localhost:21000
> select straight_join count(*)
> from
> lineitem a, lineitem b
> where
> a.l_partkey = 1 and
> a.l_orderkey = b.l_orderkey;
> -- 2020-11-11 23:12:04,385 INFO MainThread: Started query
> 314c019cd252f322:2411bc76
> -- executing against localhost:21000
> SET DEBUG_ACTION="";
> -- 2020-11-11 23:12:05,199 INFO MainThread: Started query
> 80424e68922c30f9:b2144dff
> -- executing against localhost:21000
> set debug_action="-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0";
> -- 2020-11-11 23:12:05,207 INFO MainThread: Started query
> 2a4c1f4b26ea52da:4339f3ff
> -- executing against localhost:21000
> select straight_join count(*)
> from
> lineitem a
> where
> a.l_partkey not in (select l_partkey from lineitem where l_partkey > 10)
> and a.l_partkey < 1000;
> -- 2020-11-11 23:12:05,215 INFO MainThread: Started query
> f845afd00a569446:79c5054a
> --
[jira] [Commented] (IMPALA-10339) Apparent hang in TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action
[
https://issues.apache.org/jira/browse/IMPALA-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235837#comment-17235837
]
Tim Armstrong commented on IMPALA-10339:
We're missing the "Cancelling fragment instances as directed by the
coordinator" log, which seems to be the main way that this would result in the
query getting cancelled. Not sure what's going on here.
> Apparent hang in
> TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action
> --
>
> Key: IMPALA-10339
> URL: https://issues.apache.org/jira/browse/IMPALA-10339
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build, flaky, hang
>
> Release build with this commit as the tip:
> {noformat}
> commit 9400e9b17b13f613defb6d7b9deb471256b1d95c (CDH/cdpd-master-staging)
> Author: wzhou-code
> Date: Thu Oct 29 22:32:03 2020 -0700
> IMPALA-10305: Sync Kudu's FIPS compliant changes
>
> {noformat}
> {noformat}
> Regression
> query_test.test_spilling.TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action[protocol:
> beeswax | exec_option: {'mt_dop': 0, 'default_spillable_buffer_size': '64k'}
> | table_format: parquet/none] (from pytest)
> Failing for the past 1 build (Since Failed#100 )
> Took 1 hr 59 min.
> add description
> Error Message
> query_test/test_spilling.py:134: in test_spilling_no_debug_action
> self.run_test_case('QueryTest/spilling-no-debug-action', vector)
> common/impala_test_suite.py:668: in run_test_case
> self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> common/impala_test_suite.py:485: in __verify_exceptions (expected_str,
> actual_str) E AssertionError: Unexpected exception string. Expected:
> row_regex:.*Cannot perform hash join at node with id .*. Repartitioning did
> not reduce the size of a spilled partition.* E Not found in actual: Timeout
> >7200s
> Stacktrace
> query_test/test_spilling.py:134: in test_spilling_no_debug_action
> self.run_test_case('QueryTest/spilling-no-debug-action', vector)
> common/impala_test_suite.py:668: in run_test_case
> self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> common/impala_test_suite.py:485: in __verify_exceptions
> (expected_str, actual_str)
> E AssertionError: Unexpected exception string. Expected: row_regex:.*Cannot
> perform hash join at node with id .*. Repartitioning did not reduce the size
> of a spilled partition.*
> E Not found in actual: Timeout >7200s
> Standard Error
> SET
> client_identifier=query_test/test_spilling.py::TestSpillingNoDebugActionDimensions::()::test_spilling_no_debug_action[protocol:beeswax|exec_option:{'mt_dop':0;'default_spillable_buffer_size':'64k'}|table_format:parquet/none];
> -- executing against localhost:21000
> use tpch_parquet;
> -- 2020-11-11 23:12:04,319 INFO MainThread: Started query
> c740c1c66d9679a9:6a40f161
> SET
> client_identifier=query_test/test_spilling.py::TestSpillingNoDebugActionDimensions::()::test_spilling_no_debug_action[protocol:beeswax|exec_option:{'mt_dop':0;'default_spillable_buffer_size':'64k'}|table_format:parquet/none];
> SET mt_dop=0;
> SET default_spillable_buffer_size=64k;
> -- 2020-11-11 23:12:04,320 INFO MainThread: Loading query test file:
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/testdata/workloads/functional-query/queries/QueryTest/spilling-no-debug-action.test
> -- 2020-11-11 23:12:04,323 INFO MainThread: Starting new HTTP connection
> (1): localhost
> -- executing against localhost:21000
> set debug_action="-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0";
> -- 2020-11-11 23:12:04,377 INFO MainThread: Started query
> c044afcf5ae44df9:a2e2e7c6
> -- executing against localhost:21000
> select straight_join count(*)
> from
> lineitem a, lineitem b
> where
> a.l_partkey = 1 and
> a.l_orderkey = b.l_orderkey;
> -- 2020-11-11 23:12:04,385 INFO MainThread: Started query
> 314c019cd252f322:2411bc76
> -- executing against localhost:21000
> SET DEBUG_ACTION="";
> -- 2020-11-11 23:12:05,199 INFO MainThread: Started query
> 80424e68922c30f9:b2144dff
> -- executing against localhost:21000
> set debug_action="-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0";
> -- 2020-11-11 23:12:05,207 INFO MainThread: Started query
> 2a4c1f4b26ea52da:4339f3ff
> -- executing against localhost:21000
> select straight_join count(*)
> from
> lineitem a
> where
> a.l_partkey not in (select l_partkey from lineitem where l_partkey > 10)
> and a.l_partkey < 1000;
> -- 2020-11-11 23:12:05,215 INFO MainThread: Started query
>
[jira] [Commented] (IMPALA-10339) Apparent hang in TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action
[
https://issues.apache.org/jira/browse/IMPALA-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235831#comment-17235831
]
Tim Armstrong commented on IMPALA-10339:
{noformat}
I1117 00:32:24.677520 14067 control-service.cc:142]
484209f4cc06bb05:eec7d58d] ExecQueryFInstances():
query_id=484209f4cc06bb05:eec7d58d
coord=impala-ec2-centos74-m5-4xlarge-ondemand-0d63.vpc.cloudera.com:27000
#instances=3
I1117 00:32:24.746866 18128 query-state.cc:897]
484209f4cc06bb05:eec7d58d0004] Executing instance.
instance_id=484209f4cc06bb05:eec7d58d0004 fragment_idx=3
per_fragment_instance_idx=0 coord_state_idx=1 #in-flight=34
I1117 00:32:24.767122 18132 query-state.cc:897]
484209f4cc06bb05:eec7d58d0007] Executing instance.
instance_id=484209f4cc06bb05:eec7d58d0007 fragment_idx=1
per_fragment_instance_idx=0 coord_state_idx=1 #in-flight=35
I1117 00:32:24.806087 18137 query-state.cc:897]
484209f4cc06bb05:eec7d58d0001] Executing instance.
instance_id=484209f4cc06bb05:eec7d58d0001 fragment_idx=2
per_fragment_instance_idx=0 coord_state_idx=1 #in-flight=36
I1117 00:32:25.895730 18137 krpc-data-stream-mgr.cc:337]
484209f4cc06bb05:eec7d58d0001] cancelling active streams for
fragment_instance_id=484209f4cc06bb05:eec7d58d0001
I1117 00:32:25.895849 18137 query-state.cc:906]
484209f4cc06bb05:eec7d58d0001] Instance completed.
instance_id=484209f4cc06bb05:eec7d58d0001 #in-flight=34 status=OK
I1117 00:33:07.615486 18128 krpc-data-stream-mgr.cc:337]
484209f4cc06bb05:eec7d58d0004] cancelling active streams for
fragment_instance_id=484209f4cc06bb05:eec7d58d0004
I1117 00:33:07.615584 18128 query-state.cc:906]
484209f4cc06bb05:eec7d58d0004] Instance completed.
instance_id=484209f4cc06bb05:eec7d58d0004 #in-flight=10 status=OK
I1117 00:33:08.004225 18194 krpc-data-stream-mgr.cc:306]
484209f4cc06bb05:eec7d58d0007] DeregisterRecvr():
fragment_instance_id=484209f4cc06bb05:eec7d58d0007, node=5
I1117 00:33:08.010303 14095 query-exec-mgr.cc:126] QueryState:
query_id=484209f4cc06bb05:eec7d58d refcnt=3
I1117 00:33:09.792160 18132 status.cc:88] 484209f4cc06bb05:eec7d58d0007]
Cannot perform hash join at node with id 2. Repartitioning did not reduce the
size of a spilled partition. Repartitioning level 1. Number of rows 1071394:
I1117 00:33:09.887547 18132 krpc-data-stream-mgr.cc:337]
484209f4cc06bb05:eec7d58d0007] cancelling active streams for
fragment_instance_id=484209f4cc06bb05:eec7d58d0007
I1117 00:33:09.906414 18132 krpc-data-stream-mgr.cc:306]
484209f4cc06bb05:eec7d58d0007] DeregisterRecvr():
fragment_instance_id=484209f4cc06bb05:eec7d58d0007, node=4
I1117 00:33:09.907351 18132 query-state.cc:906]
484209f4cc06bb05:eec7d58d0007] Instance completed.
instance_id=484209f4cc06bb05:eec7d58d0007 #in-flight=11
status=PARTITIONED_HASH_JOIN_REPARTITION_FAILS: Cannot perform hash join at
node with id 2. Repartitioning did not reduce the size of a spilled partition.
Repartitioning level 1. Number of rows 1071394:
I1117 00:33:09.912003 18122 query-state.cc:464]
484209f4cc06bb05:eec7d58d] UpdateBackendExecState(): last report for
484209f4cc06bb05:eec7d58d
F1117 00:33:09.914166 18122 query-state.cc:877]
484209f4cc06bb05:eec7d58d] Check failed: is_cancelled_.Load() == 1 (0
vs. 1)
{noformat}
> Apparent hang in
> TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action
> --
>
> Key: IMPALA-10339
> URL: https://issues.apache.org/jira/browse/IMPALA-10339
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build, flaky, hang
>
> Release build with this commit as the tip:
> {noformat}
> commit 9400e9b17b13f613defb6d7b9deb471256b1d95c (CDH/cdpd-master-staging)
> Author: wzhou-code
> Date: Thu Oct 29 22:32:03 2020 -0700
> IMPALA-10305: Sync Kudu's FIPS compliant changes
>
> {noformat}
> {noformat}
> Regression
> query_test.test_spilling.TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action[protocol:
> beeswax | exec_option: {'mt_dop': 0, 'default_spillable_buffer_size': '64k'}
> | table_format: parquet/none] (from pytest)
> Failing for the past 1 build (Since Failed#100 )
> Took 1 hr 59 min.
> add description
> Error Message
> query_test/test_spilling.py:134: in test_spilling_no_debug_action
> self.run_test_case('QueryTest/spilling-no-debug-action', vector)
> common/impala_test_suite.py:668: in run_test_case
> self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> common/impala_test_suite.py:485: in __verify_exceptions (expected_str,
>
[jira] [Commented] (IMPALA-10339) Apparent hang in TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action
[
https://issues.apache.org/jira/browse/IMPALA-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235825#comment-17235825
]
Tim Armstrong commented on IMPALA-10339:
I saw a crash on a very similar query that fails with repartitioning:
{noformat}
74: client_identifier (string) =
"query_test/test_spilling.py::TestSpillingDebugActionDimensions::()::test_spilling[protocol:be
eswax|exec_option:{'mt_dop':0;'debug_action':None;'default_spillable_buffer_size':'256k'}|table_format:parquet/none]",
...
I1117 00:32:24.636593 8768 Frontend.java:1522]
484209f4cc06bb05:eec7d58d] Analyzing query: select straight_join *
from lineitem l1 join lineitem l2 on l1.l_linenumber = l2.l_linenumber
where l1.l_orderkey < 10
order by l1.l_orderkey desc, l1.l_linenumber desc limit 10 db: tpch_parquet
{noformat}
This is this test from
./testdata/workloads/functional-query/queries/QueryTest/spilling.test:
{noformat}
QUERY
# Test spilling join with many duplicates in join key. We don't expect this to
succeed
# with a memory constraint: see IMPALA-4857. Limit size of probe so that query
doesn't
# bog down executing an exploding join.
# The additional "order by" and "limit" clauses make sure that a successful
# query does not too much data to the client.
set buffer_pool_limit=167m;
select straight_join *
from lineitem l1 join lineitem l2 on l1.l_linenumber = l2.l_linenumber
where l1.l_orderkey < 10
order by l1.l_orderkey desc, l1.l_linenumber desc limit 10
CATCH
Repartitioning did not reduce the size of a spilled partition
{noformat}
Backtrace on crash is:
{noformat}
F1117 00:33:09.914166 18122 query-state.cc:877]
484209f4cc06bb05:eec7d58d] Check failed: is_cancelled_.Load() == 1 (0
vs. 1)
*** Check failure stack trace: ***
@ 0x520b32c google::LogMessage::Fail()
@ 0x520cc1c google::LogMessage::SendToLog()
@ 0x520ac8a google::LogMessage::Flush()
@ 0x520e888 google::LogMessageFatal::~LogMessageFatal()
@ 0x228786f impala::QueryState::MonitorFInstances()
@ 0x2277010 impala::QueryExecMgr::ExecuteQueryHelper()
@ 0x227f924 boost::_mfi::mf1<>::operator()()
@ 0x227f1ed boost::_bi::list2<>::operator()<>()
@ 0x227e7f4 boost::_bi::bind_t<>::operator()()
@ 0x227dc0e
boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0x21438c3 boost::function0<>::operator()()
@ 0x2721a9b impala::Thread::SuperviseThread()
@ 0x2729a38 boost::_bi::list5<>::operator()<>()
@ 0x272995c boost::_bi::bind_t<>::operator()()
@ 0x272991d boost::detail::thread_data<>::run()
@ 0x3f11e61 thread_proxy
@ 0x7f460b2ffe24 start_thread
@ 0x7f4607d9634c __clone
Picked up JAVA_TOOL_OPTIONS:
-agentlib:jdwp=transport=dt_socket,address=30001,server=y,suspend=n
Wrote minidump to
/data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/ee_tests/minidumps/impalad/e3c36842-f700-42a5-575dd186-f97a0aed.dmp
{noformat}
> Apparent hang in
> TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action
> --
>
> Key: IMPALA-10339
> URL: https://issues.apache.org/jira/browse/IMPALA-10339
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build, flaky, hang
>
> Release build with this commit as the tip:
> {noformat}
> commit 9400e9b17b13f613defb6d7b9deb471256b1d95c (CDH/cdpd-master-staging)
> Author: wzhou-code
> Date: Thu Oct 29 22:32:03 2020 -0700
> IMPALA-10305: Sync Kudu's FIPS compliant changes
>
> {noformat}
> {noformat}
> Regression
> query_test.test_spilling.TestSpillingNoDebugActionDimensions.test_spilling_no_debug_action[protocol:
> beeswax | exec_option: {'mt_dop': 0, 'default_spillable_buffer_size': '64k'}
> | table_format: parquet/none] (from pytest)
> Failing for the past 1 build (Since Failed#100 )
> Took 1 hr 59 min.
> add description
> Error Message
> query_test/test_spilling.py:134: in test_spilling_no_debug_action
> self.run_test_case('QueryTest/spilling-no-debug-action', vector)
> common/impala_test_suite.py:668: in run_test_case
> self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> common/impala_test_suite.py:485: in __verify_exceptions (expected_str,
> actual_str) E AssertionError: Unexpected exception string. Expected:
> row_regex:.*Cannot perform hash join at node with id .*. Repartitioning did
> not reduce the size of a spilled partition.* E Not found in actual: Timeout
> >7200s
> Stacktrace
>
[jira] [Resolved] (IMPALA-10276) Release build sees SIGSEGV when updating the total time counter
[
https://issues.apache.org/jira/browse/IMPALA-10276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10276.
Fix Version/s: Impala 4.0
Resolution: Fixed
> Release build sees SIGSEGV when updating the total time counter
> ---
>
> Key: IMPALA-10276
> URL: https://issues.apache.org/jira/browse/IMPALA-10276
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.0
>Reporter: Joe McDonnell
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build, crash, flaky
> Fix For: Impala 4.0
>
>
> A recent release build saw an Impalad crash with the following stack:
> {noformat}
> Crash reason: SIGSEGV
> Crash address: 0x11
> #0 raise () from /lib64/libc.so.6
> #1 abort () from /lib64/libc.so.6
> #2 os::abort(bool) () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #3 VMError::report_and_die() () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #4 JVM_handle_linux_signal () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #5 signalHandler(int, siginfo*, void*) () from
> /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #6
> #7 UpdateCounter (this=0x7f93adc88030) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/util/runtime-profile-counters.h:933
> #8 impala::ScopedTimer::~ScopedTimer
> (this=0x7f93adc88030, __in_chrg=) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/util/runtime-profile-counters.h:954
> #9 impala::Coordinator::GetNext (this=this@entry=0x13ad09e00,
> results=results@entry=0xa7e18460, max_rows=max_rows@entry=-1,
> eos=eos@entry=0x7f93adc880ef,
> block_on_wait_time_us=block_on_wait_time_us@entry=0) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/runtime/coordinator.cc:864
> #10 impala::ClientRequestState::FetchRowsInternal
> (this=this@entry=0x11568000, max_rows=max_rows@entry=-1,
> fetched_rows=fetched_rows@entry=0xa7e18460,
> block_on_wait_time_us=block_on_wait_time_us@entry=0) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/service/client-request-state.cc:1090
> #11 impala::ClientRequestState::FetchRows (this=0x11568000,
> max_rows=max_rows@entry=-1, fetched_rows=fetched_rows@entry=0xa7e18460,
> block_on_wait_time_us=0) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/service/client-request-state.cc:938
> #12 impala::ImpalaServer::FetchInternal (this=this@entry=0xdd35b00,
> query_id=..., start_over=start_over@entry=false,
> fetch_size=fetch_size@entry=-1,
> query_results=query_results@entry=0x7f93adc88498) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/service/impala-beeswax-server.cc:614
> #13 impala::ImpalaServer::fetch (this=0xdd35b00, query_results=...,
> beeswax_handle=..., start_over=, fetch_size=-1) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/service/impala-beeswax-server.cc:191
> #14 beeswax::BeeswaxServiceProcessor::process_fetch (this=0x102b0c60,
> seqid=0, iprot=, oprot=0xd45d0980, callContext= out>) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/BeeswaxService.cpp:3398
> #15 beeswax::BeeswaxServiceProcessor::dispatchCall (this=0x102b0c60,
> iprot=0xd45d1e00, oprot=0xd45d0980, fname=..., seqid=0,
> callContext=0x12ea9a80) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/BeeswaxService.cpp:3200
> #16 apache::thrift::TDispatchProcessor::process (this=0x102b0c60, in=...,
> out=..., connectionContext=0x12ea9a80) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.9.3-p8/include/thrift/TDispatchProcessor.h:121
> #17 apache::thrift::server::TAcceptQueueServer::Task::run (this=0x11a92e80)
> at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/rpc/TAcceptQueueServer.cpp:84
> #18 operator() (a2=, a1=..., p=,
> this=) at
> /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/mem_fn_template.hpp:280{noformat}
> The code is updating the total time counter in the runtime profile via a
> ScopedTimer.
> This has been seen once
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
[jira] [Reopened] (IMPALA-10216) BufferPoolTest.WriteErrorBlacklistCompression is flaky on UBSAN builds
[
https://issues.apache.org/jira/browse/IMPALA-10216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reopened IMPALA-10216:
I saw this again, but unfortunately not on a branch with the logging.
> BufferPoolTest.WriteErrorBlacklistCompression is flaky on UBSAN builds
> --
>
> Key: IMPALA-10216
> URL: https://issues.apache.org/jira/browse/IMPALA-10216
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: flaky
>
> Only seen this once so far:
> {code}
> BufferPoolTest.WriteErrorBlacklistCompression
> Error Message
> Value of: FindPageInDir(pages[NO_ERROR_QUERY], error_dir) != NULL
> Actual: false
> Expected: true
> Stacktrace
> Impala/be/src/runtime/bufferpool/buffer-pool-test.cc:1764
> Value of: FindPageInDir(pages[NO_ERROR_QUERY], error_dir) != NULL
> Actual: false
> Expected: true
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10343) control_service_queue_mem_limit default is too low for large clusters
Tim Armstrong created IMPALA-10343: -- Summary: control_service_queue_mem_limit default is too low for large clusters Key: IMPALA-10343 URL: https://issues.apache.org/jira/browse/IMPALA-10343 Project: IMPALA Issue Type: Improvement Components: Distributed Exec Reporter: Tim Armstrong We've seen some cases where this queue overflows because of status reports, particularly with higher mt_dop values or large clusters. We should probably set it to a higher value by default. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-9716) Add jitter to the exponential backoff in status reporting
[ https://issues.apache.org/jira/browse/IMPALA-9716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong reassigned IMPALA-9716: - Assignee: Thomas Tauber-Marshall (was: Tim Armstrong) > Add jitter to the exponential backoff in status reporting > - > > Key: IMPALA-9716 > URL: https://issues.apache.org/jira/browse/IMPALA-9716 > Project: IMPALA > Issue Type: Improvement > Components: Distributed Exec >Affects Versions: Impala 4.0 >Reporter: Thomas Tauber-Marshall >Assignee: Thomas Tauber-Marshall >Priority: Major > Fix For: Impala 4.0 > > > When status reports fail, we use exponential backoff when retrying sending > them. However, currently the backoff is deterministic, leading to a > thundering herd problem where all of the backends for a particular query may > try to report at the same time, the coordinator is overwhelmed and rejects > some of the rpcs, then the backends all backoff by the same amount and retry > sending at the same time, leading the coordinator to be overwhelmed again. > We can help solve this by adding some random jitter to the exponential > backoff time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-9716) Add jitter to the exponential backoff in status reporting
[ https://issues.apache.org/jira/browse/IMPALA-9716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong reassigned IMPALA-9716: - Assignee: Tim Armstrong (was: Thomas Tauber-Marshall) > Add jitter to the exponential backoff in status reporting > - > > Key: IMPALA-9716 > URL: https://issues.apache.org/jira/browse/IMPALA-9716 > Project: IMPALA > Issue Type: Improvement > Components: Distributed Exec >Affects Versions: Impala 4.0 >Reporter: Thomas Tauber-Marshall >Assignee: Tim Armstrong >Priority: Major > Fix For: Impala 4.0 > > > When status reports fail, we use exponential backoff when retrying sending > them. However, currently the backoff is deterministic, leading to a > thundering herd problem where all of the backends for a particular query may > try to report at the same time, the coordinator is overwhelmed and rejects > some of the rpcs, then the backends all backoff by the same amount and retry > sending at the same time, leading the coordinator to be overwhelmed again. > We can help solve this by adding some random jitter to the exponential > backoff time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7523) Planner Test failing with "Failed to assign regions to servers after 60000 millis."
[
https://issues.apache.org/jira/browse/IMPALA-7523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-7523:
-
Assignee: (was: Tim Armstrong)
> Planner Test failing with "Failed to assign regions to servers after 6
> millis."
> ---
>
> Key: IMPALA-7523
> URL: https://issues.apache.org/jira/browse/IMPALA-7523
> Project: IMPALA
> Issue Type: Task
> Components: Frontend
>Reporter: Philip Martin
>Priority: Major
> Labels: broken-build, flaky
>
> I've seen
> {{org.apache.impala.planner.PlannerTest.org.apache.impala.planner.PlannerTest}}
> fail with the following trace:
> {code}
> java.lang.IllegalStateException: Failed to assign regions to servers after
> 6 millis.
> at
> org.apache.impala.datagenerator.HBaseTestDataRegionAssignment.performAssignment(HBaseTestDataRegionAssignment.java:153)
> at
> org.apache.impala.planner.PlannerTestBase.setUp(PlannerTestBase.java:120)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}
> I think we've seen it before as indicated in IMPALA-7061.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9121) heap-use-after-free in HdfsUtilTest CheckFilesystemsMatch
[
https://issues.apache.org/jira/browse/IMPALA-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-9121 started by Tim Armstrong.
-
> heap-use-after-free in HdfsUtilTest CheckFilesystemsMatch
> -
>
> Key: IMPALA-9121
> URL: https://issues.apache.org/jira/browse/IMPALA-9121
> Project: IMPALA
> Issue Type: Bug
>Reporter: Bikramjeet Vig
>Assignee: Tim Armstrong
>Priority: Major
> Labels: asan, broken-build, flaky-test
>
> {noformat}
> --
> 85/113 Testing: hdfs-util-test
> 85/113 Test: hdfs-util-test
> Command:
> "/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/util/hdfs-util-test"
> "-log_dir=/data/jenkins/workspace
> /impala-asf-master-core-asan/repos/Impala/logs/be_tests"
> Directory:
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util
> "hdfs-util-test" start time: Nov 01 22:21 PDT
> Output:
> --
> seed = 1572672103
> Note: Google Test filter = HdfsUtilTest.*
> [==] Running 2 tests from 1 test case.
> [--] Global test environment set-up.
> [--] 2 tests from HdfsUtilTest
> [ RUN ] HdfsUtilTest.CheckFilesystemsMatch
> 19/11/01 22:21:43 INFO util.JvmPauseMonitor: Starting JVM pause monitor
> [ OK ] HdfsUtilTest.CheckFilesystemsMatch (1012 ms)
> [ RUN ] HdfsUtilTest.CheckGetBaseName
> [ OK ] HdfsUtilTest.CheckGetBaseName (0 ms)
> [--] 2 tests from HdfsUtilTest (1012 ms total)
> [--] Global test environment tear-down
> [==] 2 tests from 1 test case ran. (1013 ms total)
> [ PASSED ] 2 tests.
> YOU HAVE 1 DISABLED TEST
> =
> ==87116==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x60362878 at pc 0x01b67bac bp 0x7fc96cf57ae0 sp 0x7fc96cf57290
> READ of size 4 at 0x60362878 thread T4
>
> Test time = 2.64 sec
> --
> Test Passed.
> "hdfs-util-test" end time: Nov 01 22:21 PDT
> "hdfs-util-test" time elapsed: 00:00:02
> --
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10316) load_nested.py failed due to out of memory during Jenkins GVO
[
https://issues.apache.org/jira/browse/IMPALA-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235095#comment-17235095
]
Tim Armstrong commented on IMPALA-10316:
I saw this again here -
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/12656/console
> load_nested.py failed due to out of memory during Jenkins GVO
> -
>
> Key: IMPALA-10316
> URL: https://issues.apache.org/jira/browse/IMPALA-10316
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Priority: Major
> Labels: broken-build, flaky
>
> The following job failed due to out of memory:
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/12588] (please click
> on "Don't keep this build forever" once this issue is resolved)
> Relevant log lines:
> {noformat}
> 02:33:42 Loading nested orc data (logging to
> /home/ubuntu/Impala/logs/data_loading/load-nested.log)...
> 02:35:39 FAILED (Took: 1 min 57 sec)
> 02:35:39 '/home/ubuntu/Impala/testdata/bin/load_nested.py -t
> tpch_nested_orc_def -f orc/def' failed. Tail of log:
> 02:35:39 2020-11-11 02:35:06,225 INFO:load_nested[348]:Executing:
> 02:35:39
> 02:35:39 CREATE EXTERNAL TABLE supplier
> 02:35:39 STORED AS orc
> 02:35:39 TBLPROPERTIES('orc.compress' =
> 'ZLIB','external.table.purge'='TRUE')
> 02:35:39 AS SELECT * FROM tmp_supplier
> 02:35:39 Traceback (most recent call last):
> 02:35:39 File "/home/ubuntu/Impala/testdata/bin/load_nested.py", line 415,
> in
> 02:35:39 load()
> 02:35:39 File "/home/ubuntu/Impala/testdata/bin/load_nested.py", line 349,
> in load
> 02:35:39 hive.execute(stmt)
> 02:35:39 File "/home/ubuntu/Impala/tests/comparison/db_connection.py", line
> 206, in execute
> 02:35:39 return self._cursor.execute(sql, *args, **kwargs)
> 02:35:39 File
> "/home/ubuntu/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/hiveserver2.py",
> line 331, in execute
> 02:35:39 self._wait_to_finish() # make execute synchronous
> 02:35:39 File
> "/home/ubuntu/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/hiveserver2.py",
> line 413, in _wait_to_finish
> 02:35:39 raise OperationalError(resp.errorMessage)
> 02:35:39 impala.error.OperationalError: Error while compiling statement:
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
> vertexId=vertex_1605060173780_0039_2_00, diagnostics=[Task failed,
> taskId=task_1605060173780_0039_2_00_00, diagnostics=[TaskAttempt 0
> failed, info=[Container container_1605060173780_0039_01_02 finished with
> diagnostics set to [Container failed, exitCode=-104. [2020-11-11
> 02:35:11.768]Container
> [pid=16810,containerID=container_1605060173780_0039_01_02] is running
> 7729152B beyond the 'PHYSICAL' memory limit. Current usage: 1.0 GB of 1 GB
> physical memory used; 2.5 GB of 2.1 GB virtual memory used. Killing
> container.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9121) heap-use-after-free in HdfsUtilTest CheckFilesystemsMatch
[
https://issues.apache.org/jira/browse/IMPALA-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235089#comment-17235089
]
Tim Armstrong commented on IMPALA-9121:
---
I'm suspicious about how it's using ExecEnv - that starts up a bunch of
background threads that might be the ones causing the problem here during
teardown. I think I can minimize that.
> heap-use-after-free in HdfsUtilTest CheckFilesystemsMatch
> -
>
> Key: IMPALA-9121
> URL: https://issues.apache.org/jira/browse/IMPALA-9121
> Project: IMPALA
> Issue Type: Bug
>Reporter: Bikramjeet Vig
>Assignee: Tim Armstrong
>Priority: Major
> Labels: asan, broken-build, flaky-test
>
> {noformat}
> --
> 85/113 Testing: hdfs-util-test
> 85/113 Test: hdfs-util-test
> Command:
> "/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/util/hdfs-util-test"
> "-log_dir=/data/jenkins/workspace
> /impala-asf-master-core-asan/repos/Impala/logs/be_tests"
> Directory:
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util
> "hdfs-util-test" start time: Nov 01 22:21 PDT
> Output:
> --
> seed = 1572672103
> Note: Google Test filter = HdfsUtilTest.*
> [==] Running 2 tests from 1 test case.
> [--] Global test environment set-up.
> [--] 2 tests from HdfsUtilTest
> [ RUN ] HdfsUtilTest.CheckFilesystemsMatch
> 19/11/01 22:21:43 INFO util.JvmPauseMonitor: Starting JVM pause monitor
> [ OK ] HdfsUtilTest.CheckFilesystemsMatch (1012 ms)
> [ RUN ] HdfsUtilTest.CheckGetBaseName
> [ OK ] HdfsUtilTest.CheckGetBaseName (0 ms)
> [--] 2 tests from HdfsUtilTest (1012 ms total)
> [--] Global test environment tear-down
> [==] 2 tests from 1 test case ran. (1013 ms total)
> [ PASSED ] 2 tests.
> YOU HAVE 1 DISABLED TEST
> =
> ==87116==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x60362878 at pc 0x01b67bac bp 0x7fc96cf57ae0 sp 0x7fc96cf57290
> READ of size 4 at 0x60362878 thread T4
>
> Test time = 2.64 sec
> --
> Test Passed.
> "hdfs-util-test" end time: Nov 01 22:21 PDT
> "hdfs-util-test" time elapsed: 00:00:02
> --
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9121) heap-use-after-free in HdfsUtilTest CheckFilesystemsMatch
[
https://issues.apache.org/jira/browse/IMPALA-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235070#comment-17235070
]
Tim Armstrong commented on IMPALA-9121:
---
This does reproduce after looping for a while, I guess it's happening after the
tests finish and are torn down.
{noformat}
Note: Google Test filter = HdfsUtilTest.*
[==] Running 2 tests from 1 test case.
[--] Global test environment set-up.
[--] 2 tests from HdfsUtilTest
[ RUN ] HdfsUtilTest.CheckFilesystemsMatch
20/11/18 15:59:03 INFO util.JvmPauseMonitor: Starting JVM pause monitor
[ OK ] HdfsUtilTest.CheckFilesystemsMatch (935 ms)
[ RUN ] HdfsUtilTest.CheckGetBaseName
[ OK ] HdfsUtilTest.CheckGetBaseName (0 ms)
[--] 2 tests from HdfsUtilTest (935 ms total)
[--] Global test environment tear-down
[==] 2 tests from 1 test case ran. (935 ms total)
[ PASSED ] 2 tests.
YOU HAVE 1 DISABLED TEST
=
==31924==ERROR: AddressSanitizer: heap-use-after-free on address 0x6070ead0
at pc 0x01
e34a4b bp 0x7fa2e44ffac0 sp 0x7fa2e44ff270
READ of size 5 at 0x6070ead0 thread T4
{noformat}
> heap-use-after-free in HdfsUtilTest CheckFilesystemsMatch
> -
>
> Key: IMPALA-9121
> URL: https://issues.apache.org/jira/browse/IMPALA-9121
> Project: IMPALA
> Issue Type: Bug
>Reporter: Bikramjeet Vig
>Assignee: Tim Armstrong
>Priority: Major
> Labels: asan, broken-build, flaky-test
>
> {noformat}
> --
> 85/113 Testing: hdfs-util-test
> 85/113 Test: hdfs-util-test
> Command:
> "/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/util/hdfs-util-test"
> "-log_dir=/data/jenkins/workspace
> /impala-asf-master-core-asan/repos/Impala/logs/be_tests"
> Directory:
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util
> "hdfs-util-test" start time: Nov 01 22:21 PDT
> Output:
> --
> seed = 1572672103
> Note: Google Test filter = HdfsUtilTest.*
> [==] Running 2 tests from 1 test case.
> [--] Global test environment set-up.
> [--] 2 tests from HdfsUtilTest
> [ RUN ] HdfsUtilTest.CheckFilesystemsMatch
> 19/11/01 22:21:43 INFO util.JvmPauseMonitor: Starting JVM pause monitor
> [ OK ] HdfsUtilTest.CheckFilesystemsMatch (1012 ms)
> [ RUN ] HdfsUtilTest.CheckGetBaseName
> [ OK ] HdfsUtilTest.CheckGetBaseName (0 ms)
> [--] 2 tests from HdfsUtilTest (1012 ms total)
> [--] Global test environment tear-down
> [==] 2 tests from 1 test case ran. (1013 ms total)
> [ PASSED ] 2 tests.
> YOU HAVE 1 DISABLED TEST
> =
> ==87116==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x60362878 at pc 0x01b67bac bp 0x7fc96cf57ae0 sp 0x7fc96cf57290
> READ of size 4 at 0x60362878 thread T4
>
> Test time = 2.64 sec
> --
> Test Passed.
> "hdfs-util-test" end time: Nov 01 22:21 PDT
> "hdfs-util-test" time elapsed: 00:00:02
> --
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-9121) heap-use-after-free in HdfsUtilTest CheckFilesystemsMatch
[
https://issues.apache.org/jira/browse/IMPALA-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-9121:
-
Assignee: Tim Armstrong
> heap-use-after-free in HdfsUtilTest CheckFilesystemsMatch
> -
>
> Key: IMPALA-9121
> URL: https://issues.apache.org/jira/browse/IMPALA-9121
> Project: IMPALA
> Issue Type: Bug
>Reporter: Bikramjeet Vig
>Assignee: Tim Armstrong
>Priority: Major
> Labels: asan, broken-build, flaky-test
>
> {noformat}
> --
> 85/113 Testing: hdfs-util-test
> 85/113 Test: hdfs-util-test
> Command:
> "/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/util/hdfs-util-test"
> "-log_dir=/data/jenkins/workspace
> /impala-asf-master-core-asan/repos/Impala/logs/be_tests"
> Directory:
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util
> "hdfs-util-test" start time: Nov 01 22:21 PDT
> Output:
> --
> seed = 1572672103
> Note: Google Test filter = HdfsUtilTest.*
> [==] Running 2 tests from 1 test case.
> [--] Global test environment set-up.
> [--] 2 tests from HdfsUtilTest
> [ RUN ] HdfsUtilTest.CheckFilesystemsMatch
> 19/11/01 22:21:43 INFO util.JvmPauseMonitor: Starting JVM pause monitor
> [ OK ] HdfsUtilTest.CheckFilesystemsMatch (1012 ms)
> [ RUN ] HdfsUtilTest.CheckGetBaseName
> [ OK ] HdfsUtilTest.CheckGetBaseName (0 ms)
> [--] 2 tests from HdfsUtilTest (1012 ms total)
> [--] Global test environment tear-down
> [==] 2 tests from 1 test case ran. (1013 ms total)
> [ PASSED ] 2 tests.
> YOU HAVE 1 DISABLED TEST
> =
> ==87116==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x60362878 at pc 0x01b67bac bp 0x7fc96cf57ae0 sp 0x7fc96cf57290
> READ of size 4 at 0x60362878 thread T4
>
> Test time = 2.64 sec
> --
> Test Passed.
> "hdfs-util-test" end time: Nov 01 22:21 PDT
> "hdfs-util-test" time elapsed: 00:00:02
> --
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9965) test_result_spooling seems to be still flaky under asan
[
https://issues.apache.org/jira/browse/IMPALA-9965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-9965.
---
Resolution: Cannot Reproduce
I looped for a few hours locally and no luck. Haven't seen it happen recently.
Reopen and attach coordinator logs if it reoccurs.
> test_result_spooling seems to be still flaky under asan
> ---
>
> Key: IMPALA-9965
> URL: https://issues.apache.org/jira/browse/IMPALA-9965
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
>
> {noformat}
> Error Message
> query_test/test_result_spooling.py:98: in test_spilling .format(query,
> timeout)) E Timeout: Query select * from functional.alltypes order by id
> limit 1500 did not spill spooled results within the timeout 30
> Stacktrace
> query_test/test_result_spooling.py:98: in test_spilling
> .format(query, timeout))
> E Timeout: Query select * from functional.alltypes order by id limit 1500
> did not spill spooled results within the timeout 30
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10111) TestWebPage::test_query_stmt is flaky
[
https://issues.apache.org/jira/browse/IMPALA-10111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234955#comment-17234955
]
Tim Armstrong commented on IMPALA-10111:
I wonder if it just aged out of the query logs and could be fixed by increasing
the query log size...
Anyway I'm not having luck reproducing it so I'll close it out.
If it reoccurs we should attach logs
> TestWebPage::test_query_stmt is flaky
> -
>
> Key: IMPALA-10111
> URL: https://issues.apache.org/jira/browse/IMPALA-10111
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Assignee: Tim Armstrong
>Priority: Major
> Labels: broken-build, flaky
>
> Found in an unrelated patch:
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3032/testReport/junit/webserver.test_web_pages/TestWebPage/test_query_stmt/
> {code:java}
> webserver/test_web_pages.py:418: in test_query_stmt
> assert check_if_contains, "No matching statement found in the jsons."
> E AssertionError: No matching statement found in the jsons.
> E assert False {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10111) TestWebPage::test_query_stmt is flaky
[
https://issues.apache.org/jira/browse/IMPALA-10111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10111.
Resolution: Cannot Reproduce
> TestWebPage::test_query_stmt is flaky
> -
>
> Key: IMPALA-10111
> URL: https://issues.apache.org/jira/browse/IMPALA-10111
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Assignee: Tim Armstrong
>Priority: Major
> Labels: broken-build, flaky
>
> Found in an unrelated patch:
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/3032/testReport/junit/webserver.test_web_pages/TestWebPage/test_query_stmt/
> {code:java}
> webserver/test_web_pages.py:418: in test_query_stmt
> assert check_if_contains, "No matching statement found in the jsons."
> E AssertionError: No matching statement found in the jsons.
> E assert False {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10293) Improvement to TestParquet.test_multiple_blocks_mt_dop
[
https://issues.apache.org/jira/browse/IMPALA-10293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10293.
Resolution: Not A Bug
It looks like this failure occurred on a test run where an impalad crashed,
hence the difference in distribution across nodes:
{noformat}
Error Message
Minidump generated:
/data/jenkins/workspace/impala-asf-master-exhaustive/repos/Impala/logs/ee_tests/minidumps/impalad/20d30f8e-cb1c-46fd-10f827af-c504e9ab.dmp
Standard Error
Operating system: Linux
0.0.0 Linux 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20
20:32:50 UTC 2017 x86_64
CPU: amd64
family 6 model 85 stepping 4
1 CPU
GPU: UNKNOWN
Crash reason: SIGSEGV
Crash address: 0x2a
Process uptime: not available
Thread 406 (crashed)
0 impalad!tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) +
0x3b
rax = 0x0001 rdx = 0x7ffa171cd220
rcx = 0x002a rbx = 0x002a
rsi = 0x0001 rdi = 0x07721d00
rbp = 0x7ffa171cd180 rsp = 0x7ffa171cd150
r8 = 0x07721d40r9 = 0x7ffa171cd378
r10 = 0x7ffaff2db8c0 r11 = 0x7ffafe802cd0
r12 = 0x07721d00 r13 = 0x09255a00
r14 = 0x7ffa171cd228 r15 = 0x09093140
rip = 0x05331a3b
Found by: given as instruction pointer in context
1 impalad!tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**,
void**) + 0x1c
rbx = 0x07721d00 rbp = 0x7ffa171cd1b0
rsp = 0x7ffa171cd190 r12 = 0x0001
r13 = 0x7ffa171cd220 r14 = 0x7ffa171cd228
r15 = 0x09093140 rip = 0x05331d1c
Found by: call frame info
2 impalad!tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) + 0xaf
rbx = 0x090933e0 rbp = 0x7ffa171cd210
rsp = 0x7ffa171cd1c0 r12 = 0x07721d00
r13 = 0x0001 r14 = 0x7ffa171cd220
r15 = 0x09093140 rip = 0x05331e0f
Found by: call frame info
3 impalad!tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long,
unsigned long) + 0x5d
rbx = 0x090933e0 rbp = 0x7ffa171cd260
rsp = 0x7ffa171cd220 r12 = 0x0240
r13 = 0x0071 r14 = 0x0270
r15 = 0x09093140 rip = 0x053417ad
Found by: call frame info
4 impalad!tc_newarray + 0x25b
rbx = 0x09093140 rbp = 0x7ffa171cd2a0
rsp = 0x7ffa171cd270 r12 = 0x0201
r13 = 0x0240 r14 = 0x7ffa171cd387
r15 = 0x rip = 0x0547837b
Found by: call frame info
5 libstdc++.so.6.0.24 + 0x11f5cd
rbx = 0x7ffa171cd2e0 rbp = 0x7ffa171cd2f0
rsp = 0x7ffa171cd2b0 r12 = 0x7ffa171cd2f0
r13 = 0x0010 r14 = 0x7ffa171cd387
r15 = 0x rip = 0x7ffaff07a5cd
Found by: call frame info
6 libstdc++.so.6.0.24 + 0x112e0b
rbp = 0x7ffa171cd2f0 rsp = 0x7ffa171cd2e0
rip = 0x7ffaff06de0b
Found by: stack scanning
7 libstdc++.so.6.0.24 + 0x11db89
rsp = 0x7ffa171cd330 rip = 0x7ffaff078b89
Found by: stack scanning
8 libstdc++.so.6.0.24 + 0x102bd3
rsp = 0x7ffa171cd360 rip = 0x7ffaff05dbd3
Found by: stack scanning
9 libstdc++.so.6.0.24 + 0x1029e9
rsp = 0x7ffa171cd370 rip = 0x7ffaff05d9e9
Found by: stack scanning
10 libstdc++.so.6.0.24 + 0x102b05
rsp = 0x7ffa171cd3a0 rip = 0x7ffaff05db05
Found by: stack scanning
11 libstdc++.so.6.0.24 + 0x10e3b5
rsp = 0x7ffa171cd420 rip = 0x7ffaff0693b5
Found by: stack scanning
12 impalad!impala::PrintId(impala::TUniqueId const&,
std::__cxx11::basic_string, std::allocator >
const&) [debug-util.cc : 109 + 0x19]
rsp = 0x7ffa171cd470 rip = 0x0264b193
Found by: stack scanning
Thread 0
0 libpthread-2.17.so + 0xb945
rax = 0xfe00 rdx = 0x0001
rcx = 0x rbx = 0x
rsi = 0x0080 rdi = 0x139d1444
rbp = 0x7ffebf6bef60 rsp = 0x7ffebf6beea0
r8 = 0x139d1400r9 = 0x
r10 = 0x r11 = 0x0246
r12 = 0x12a7a500 r13 = 0x7ffebf6bf8f0
r14 = 0x r15 = 0x
rip = 0x7ffb01ce4945
Found by: given as instruction pointer in context
1 impalad!boost::thread::join_noexcept() + 0x54
rbp = 0x7ffebf6bef80 rsp = 0x7ffebf6bef70
rip = 0x03f375d4
Found by: previous frame's frame pointer
2 impalad!boost::thread::join() [thread.hpp : 766 + 0xc]
rbx = 0x0001 rbp = 0x7ffebf6bf030
rsp = 0x7ffebf6befc0 r12 = 0x0ea55538
r13 = 0x7ffebf6bf8f0
[jira] [Commented] (IMPALA-9058) S3 tests failing with FileNotFoundException getVersionMarkerItem on ../VERSION
[
https://issues.apache.org/jira/browse/IMPALA-9058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234949#comment-17234949
]
Tim Armstrong commented on IMPALA-9058:
---
I think what I need to do next is find an instance of this and dig through the
logs to trace the full lifecycle of the table with the problem, i.e. understand
how/when was loaded, when it was accessed by any queries, etc. Maybe that will
provide some clues.
> S3 tests failing with FileNotFoundException getVersionMarkerItem on ../VERSION
> --
>
> Key: IMPALA-9058
> URL: https://issues.apache.org/jira/browse/IMPALA-9058
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Sahil Takiar
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
>
> I've seen this happen several times now, S3 tests intermittently fail with an
> error such as:
> {code:java}
> Query aborted:InternalException: Error adding partitions E CAUSED BY:
> MetaException: java.io.IOException: Got exception:
> java.io.FileNotFoundException getVersionMarkerItem on ../VERSION:
> com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException: Requested
> resource not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
> ResourceNotFoundException; Request ID:
> 8T9IS939MDI7ASOB0IJCC34J3NVV4KQNSO5AEMVJF66Q9ASUAAJG) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
