Gabor Kaszab created IMPALA-8782:
------------------------------------
Summary: Impala errors can make profiles huge
Key: IMPALA-8782
URL: https://issues.apache.org/jira/browse/IMPALA-8782
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.9.0
Reporter: Gabor Kaszab
We observed that in some cases Impala could flood the query profile with error
messages causing other systems that read the profiles crash (e.g. Cloudera
Manager) due to the size of the profile. Let's consider limiting the number of
errors we write into the profile.
In the particular case I saw the profile was full of these errors, one of this
for each row:
{code:java}
Error converting column: 4 to TIMESTAMP\nError converting column: 10 to
INT\nError converting column: 11 to INT\nError converting column: 13 to
INT\nError converting column: 15 to INT\nError converting column: 16 to
INT\nError converting column: 19 to INT\nError converting column: 20 to
INT\nError converting column: 21 to INT\nError converting column: 22 to
INT\nError converting column: 23 to INT\nError converting column: 24 to
INT\nError converting column: 26 to INT\nError converting column: 29 to
INT\nError converting column: 31 to INT\nError converting column: 35 to
INT\nError converting column: 36 to INT\nError converting column: 37 to
INT\nError converting column: 39 to INT\nError converting column: 40 to
INT\nError converting column: 41 to INT\nError converting column: 42 to
INT\nError converting column: 43 to INT\nError converting column: 44 to
INT\nError converting column: 45 to INT\nError converting column: 46 to
INT\nError converting column: 48 to INT\nError converting column: 50 to
INT\nError converting column: 51 to INT\nError converting column: 53 to
INT\nError converting column: 54 to INT\nError converting column: 59 to
INT\nError converting column: 60 to INT\nError converting column: 63 to
INT\nError converting column: 64 to INT\nError converting column: 65 to
INT\nError converting column: 69 to INT\nError converting column: 70 to
INT\nError converting column: 71 to INT\nError converting column: 72 to
INT\nError converting column: 73 to INT\nError converting column: 75 to
INT\nError converting column: 76 to INT\nError converting column: 77 to
INT\nError converting column: 78 to INT\nError converting column: 79 to
INT\nError converting column: 80 to INT\nError converting column: 81 to
INT\nError converting column: 82 to INT\nError converting column: 83 to
INT\nError converting column: 84 to INT\nError converting column: 85 to
INT\nError converting column: 87 to INT\nError converting column: 88 to
INT\nError converting column: 90 to INT\nError converting column: 91 to
INT\nError converting column: 95 to INT\nError converting column: 96 to
INT\nError converting column: 97 to INT\nError converting column: 99 to
INT\nError converting column: 102 to INT\nError converting column: 103 to
INT\nError converting column: 104 to INT\nError converting column: 106 to
INT\nError converting column: 107 to INT\nError converting column: 109 to
INT\nError converting column: 117 to INT\nError converting column: 119 to
INT\nError parsing row: file:
hdfs://nameserviceci/data/CTL/encrypt/db/ingest/raw/capm/file=interfaces/dt=20170626/interfaces.1498453200039.snappy,
before offset: 43276630\n
{code}
As a result, the uncompressed profile grew something like 700MB.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
