[
https://issues.apache.org/jira/browse/IMPALA-10808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17394416#comment-17394416
]
ASF subversion and git services commented on IMPALA-10808:
----------------------------------------------------------
Commit 599c84b4dd18c591df59f515245305636b2bc5ec in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=599c84b ]
IMPALA-10808: (addendum) Abort on illegal decimal parquet schemas
The previous patch added checks on illegal decimal schemas of parquet
files. However, it doesn't return a non-ok status in
ParquetMetadataUtils::ValidateColumn if abort_on_error is set to false.
So we continue to use the illegal file schema and hit the DCHECK.
This patch fixes this and adding test coverage for illegal decimal
schemas.
Tests:
- Add a bad parquet file with illegal decimal schemas.
- Add e2e tests on the file.
- Ran test_fuzz_decimal_tbl 100 times. Saw the errors are caught as
expected.
Change-Id: I623f255a7f40be57bfa4ade98827842cee6f1fee
Reviewed-on: http://gerrit.cloudera.org:8080/17748
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Crash of illegal decimal schema in test_fuzz_decimal_tbl
> --------------------------------------------------------
>
> Key: IMPALA-10808
> URL: https://issues.apache.org/jira/browse/IMPALA-10808
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.1.0
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Blocker
> Labels: broken-build
> Fix For: Impala 4.1.0
>
>
> Recently saw two unrelated jobs failed by the same crash:
> * [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/14369]
> * [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/14381]
> For example in the second job, the test that crashes impalad is {code}
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::()::test_fuzz_decimal_tbl[protocol:beeswax|exec_option:{'debug_action':'-1:OPEN:[email protected]';'abort_on_error':False;'mem_limit':'512m';'num_nodes':0}|table_format:parquet/none
> {code}
> The failure is
> {code:java}
> I0720 03:34:53.168516 126039 runtime-state.cc:196]
> 8a42e69ff49106c8:d2096a7000000001] Error from query
> 8a42e69ff49106c8:d2096a7000000000: File
> 'hdfs://localhost:20500/test-warehouse/test_fuzz_decimal_tbl_4a8e12be.db/decimal_tbl/d6=1/copy1_6b48619353a75ffb-66460f7400000000_973668612_data.0.parq'
> column 'd1' does not have the decimal precision set.
> F0720 03:34:53.168567 126039 types.h:282] 8a42e69ff49106c8:d2096a7000000001]
> Check failed: precision > 0 (0 vs. 0)
> {code}
> CC [~boroknagyz] who owns the first job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
