[jira] [Commented] (IMPALA-6975) TestRuntimeRowFilters.test_row_filters failing with Memory limit exceeded
[
https://issues.apache.org/jira/browse/IMPALA-6975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464243#comment-16464243
]
Tim Armstrong commented on IMPALA-6975:
---
A MEM_LIMIT probably just needs to be adjusted...
> TestRuntimeRowFilters.test_row_filters failing with Memory limit exceeded
> -
>
> Key: IMPALA-6975
> URL: https://issues.apache.org/jira/browse/IMPALA-6975
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.0
>Reporter: Sailesh Mukil
>Priority: Critical
> Labels: build-failure
>
> {code:java}
> Error Message
> query_test/test_runtime_filters.py:171: in test_row_filters
> test_file_vars={'$RUNTIME_FILTER_WAIT_TIME_MS' : str(WAIT_TIME_MS)})
> common/impala_test_suite.py:405: in run_test_case result =
> self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:620: in __execute_query return
> impalad_client.execute(query, user=user) common/impala_connection.py:160: in
> execute return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute handle =
> self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:341: in __execute_query
> self.wait_for_completion(handle) beeswax/impala_beeswax.py:361: in
> wait_for_completion raise ImpalaBeeswaxException("Query aborted:" +
> error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E
> Query aborted:Memory limit exceeded: ParquetColumnReader::ReadDataPage()
> failed to allocate 65533 bytes for decompressed data. E HDFS_SCAN_NODE
> (id=0) could not allocate 64.00 KB without exceeding limit. E Error
> occurred on backend
> impala-boost-static-burst-slave-el7-1aa3.vpc.cloudera.com:22001 by fragment
> f9423526590ed30b:732f1db10001 E Memory left in process limit: 11.14 GB
> E Memory left in query limit: 22.16 KB E
> Query(f9423526590ed30b:732f1db1): Limit=200.00 MB Reservation=160.00
> MB ReservationLimit=160.00 MB OtherMemory=39.98 MB Total=199.98 MB
> Peak=199.98 MB E Fragment f9423526590ed30b:732f1db10007:
> Reservation=134.00 MB OtherMemory=30.22 MB Total=164.22 MB Peak=164.22 MB E
> Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB
> OtherMemory=0 Total=2.00 MB Peak=2.00 MB E AGGREGATION_NODE (id=3):
> Total=4.00 KB Peak=4.00 KB E Exprs: Total=4.00 KB Peak=4.00 KB E
> HASH_JOIN_NODE (id=2): Reservation=132.00 MB OtherMemory=198.25 KB
> Total=132.19 MB Peak=132.19 MB E Exprs: Total=25.12 KB Peak=25.12 KB
> E Hash Join Builder (join_node_id=2): Total=157.12 KB Peak=157.12 KB
> E Hash Join Builder (join_node_id=2) Exprs: Total=149.12 KB
> Peak=149.12 KB E EXCHANGE_NODE (id=4): Reservation=14.91 MB
> OtherMemory=67.76 KB Total=14.97 MB Peak=14.97 MB E KrpcDeferredRpcs:
> Total=67.76 KB Peak=67.76 KB E EXCHANGE_NODE (id=5): Reservation=15.03
> MB OtherMemory=0 Total=15.03 MB Peak=15.03 MB E KrpcDeferredRpcs:
> Total=0 Peak=45.12 KB E KrpcDataStreamSender (dst_id=6): Total=16.00 KB
> Peak=16.00 KB E Fragment f9423526590ed30b:732f1db10001:
> Reservation=26.00 MB OtherMemory=9.75 MB Total=35.75 MB Peak=35.75 MB E
> Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB
> OtherMemory=0 Total=2.00 MB Peak=2.00 MB E HDFS_SCAN_NODE (id=0):
> Reservation=24.00 MB OtherMemory=9.30 MB Total=33.30 MB Peak=33.30 MB E
> Exprs: Total=260.00 KB Peak=260.00 KB E KrpcDataStreamSender
> (dst_id=4): Total=426.57 KB Peak=458.57 KB E KrpcDataStreamSender
> (dst_id=4) Exprs: Total=256.00 KB Peak=256.00 KB E Fragment
> f9423526590ed30b:732f1db10004: Reservation=0 OtherMemory=0 Total=0
> Peak=29.59 MB E HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0
> Total=0 Peak=29.32 MB E KrpcDataStreamSender (dst_id=5): Total=0
> Peak=266.57 KB EE Memory limit exceeded:
> ParquetColumnReader::ReadDataPage() failed to allocate 65533 bytes for
> decompressed data. E HDFS_SCAN_NODE (id=0) could not allocate 64.00 KB
> without exceeding limit. E Error occurred on backend
> impala-boost-static-burst-slave-el7-1aa3.vpc.cloudera.com:22001 by fragment
> f9423526590ed30b:732f1db10001 E Memory left in process limit: 11.14 GB
> E Memory left in query limit: 22.16 KB E
> Query(f9423526590ed30b:732f1db1): Limit=200.00 MB Reservation=160.00
> MB ReservationLimit=160.00 MB OtherMemory=39.98 MB Total=199.98 MB
> Peak=199.98 MB E Fragment f9423526590ed30b:732f1db10007:
> Reservation=134.00 MB OtherMemory=30.22 MB Total=164.22 MB Peak=164.22 MB E
> Runtime Filter Bank:
[jira] [Resolved] (IMPALA-6507) Consider removing --disable_mem_pools debugging feature
[ https://issues.apache.org/jira/browse/IMPALA-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-6507. --- Resolution: Fixed Fix Version/s: Impala 3.1.0 Impala 2.13.0 > Consider removing --disable_mem_pools debugging feature > --- > > Key: IMPALA-6507 > URL: https://issues.apache.org/jira/browse/IMPALA-6507 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: resource-management > Fix For: Impala 2.13.0, Impala 3.1.0 > > > We should consider removing support for the --disable_mem_pools feature. It > was originally somewhat useful for debugging memory-related issues where > memory pooling could limit the effectiveness of ASAN's checks. However, now > we use ASAN's poisoning in all cases, which provides most of the same > coverage and is used by default with ASAN. > We don't routinely test --disable_mem_pools so we should consider removing it > to save the burden of maintaining these extra code paths. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-6227) TestAdmissionControllerStress can be flaky
[
https://issues.apache.org/jira/browse/IMPALA-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464480#comment-16464480
]
Tim Armstrong edited comment on IMPALA-6227 at 5/4/18 11:04 PM:
Attached logs. I think this is the interesting part:
{noformat}
MainThread: Main loop, curr_metrics: admitted=26, queued=20, dequeued=14,
rejected=20, released=18, timed-out=0
MainThread: Main loop, will request 6 queries to end
MainThread: Requesting 6 clients to end queries
MainThread: Cancelling query 2
-- fetching results from:
MainThread: Cancelling query 14
MainThread: Cancelling query 1
MainThread: Cancelling query 31
MainThread: Cancelling query 19
MainThread: Cancelling query 23
Thread-689: Ending query
QueryHandle(log_context='c04a7c92333c0646:63dfc09c',
id='c04a7c92333c0646:63dfc09c') by CLIENT_CLOSE
-- closing query for operation handle:
Thread-697: Ending query
QueryHandle(log_context='4f4ce9ddfea34b66:4788b641',
id='4f4ce9ddfea34b66:4788b641') by CLIENT_CLOSE
-- closing query for operation handle:
Thread-667: Ending query
QueryHandle(log_context='7e474d8a33622077:2e7aacc8',
id='7e474d8a33622077:2e7aacc8') by CLIENT_CANCEL
-- canceling operation:
-- fetching results from:
Thread-680: Ending query
QueryHandle(log_context='364771ca11f54d6f:d826796e',
id='364771ca11f54d6f:d826796e') by QUERY_TIMEOUT
-- getting state for operation:
Thread-685: Ending query
QueryHandle(log_context='b84cd4660695d218:b2b187d6',
id='b84cd4660695d218:b2b187d6') by CLIENT_CLOSE
-- closing query for operation handle:
-- fetching results from:
Thread-668: Ending query
QueryHandle(log_context='f04dcf86fc9b574c:86fcff57',
id='f04dcf86fc9b574c:86fcff57') by QUERY_TIMEOUT
-- getting state for operation:
Thread-677: Admitted query 11
Thread-700: Admitted query 34
-- fetching results from:
Thread-695: Admitted query 29
Thread-679: Admitted query 13
-- getting state for operation:
-- getting state for operation:
-- getting state for operation:
-- getting state for operation:
-- getting state for operation:
-- getting state for operation:
Thread-671: Admitted query 5
Thread-676: Admitted query 10
-- fetching results from:
MainThread: wait_for_metric_changes, initial=admitted=26, queued=20,
dequeued=14, rejected=20, released=18, timed-out=0
MainThread: wait_for_metric_changes, current=admitted=30, queued=20,
dequeued=20, rejected=20, released=24, timed-out=0
MainThread: DeltaSum=6 Deltas={'dequeued': 6, 'admitted': 4, 'released': 6,
'rejected': 0, 'queued': 0, 'timed-out': 0} (Expected=6 for
metrics=['released'])
MainThread: Found all 6 metrics after 0.5 seconds
MainThread: wait_for_metric_changes, initial=admitted=26, queued=20,
dequeued=14, rejected=20, released=18, timed-out=0
MainThread: wait_for_metric_changes, current=admitted=30, queued=20,
dequeued=20, rejected=20, released=24, timed-out=0
MainThread: DeltaSum=4 Deltas={'dequeued': 6, 'admitted': 4, 'released': 6,
'rejected': 0, 'queued': 0, 'timed-out': 0} (Expected=5 for
metrics=['admitted', 'timed-out'])
MainThread: wait_for_metric_changes, current=admitted=30, queued=20,
dequeued=20, rejected=20, released=24, timed-out=0
MainThread: DeltaSum=4 Deltas={'dequeued': 6, 'admitted': 4, 'released': 6,
'rejected': 0, 'queued': 0, 'timed-out': 0} (Expected=5 for
metrics=['admitted', 'timed-out'])
MainThread: wait_for_metric_changes, current=admitted=30, queued=20,
dequeued=20, rejected=20, released=24, timed-out=0
MainThread: DeltaSum=4 Deltas={'dequeued': 6, 'admitted': 4, 'released': 6,
'rejected': 0, 'queued': 0, 'timed-out': 0} (Expected=5 for
metrics=['admitted', 'timed-out'])
MainThread: wait_for_metric_changes, current=admitted=30, queued=20,
dequeued=20, rejected=20, released=24, timed-out=0
MainThread: DeltaSum=4 Deltas={'dequeued': 6, 'admitted': 4, 'released': 6,
'rejected': 0, 'queued': 0, 'timed-out': 0} (Expected=5 for
metrics=['admitted', 'timed-out'])
MainThread: wait_for_metric_changes, current=admitted=30, queued=20,
dequeued=20, rejected=20, released=24, timed-out=0
MainThread: DeltaSum=4 Deltas={'dequeued': 6, 'admitted': 4, 'released': 6,
'rejected': 0, 'queued': 0, 'timed-out': 0} (Expected=5 for
metrics=['admitted', 'timed-out'])
MainThread: wait_for_metric_changes, current=admitted=30, queued=20,
dequeued=20, rejected=20, released=24, timed-out=0
MainThread: DeltaSum=4 Deltas={'dequeued': 6, 'admitted': 4, 'released': 6,
'rejected': 0, 'queued': 0, 'timed-out': 0} (Expected=5 for
metrics=['admitted', 'timed-out'])
{noformat}
was (Author: tarmstrong):
Attached logs. I think this is the interesting part:
{noformat}
Thread-676: Admitted query 10
-- fetching results from:
MainThread: wait_for_metric_changes, initial=admitted=26, queued=20,
dequeued=14, rejected=20, released=18, timed-out=0
MainThread:
[jira] [Updated] (IMPALA-6227) TestAdmissionControllerStress can be flaky
[
https://issues.apache.org/jira/browse/IMPALA-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-6227:
--
Attachment: TEST-impala-custom-cluster.xml
> TestAdmissionControllerStress can be flaky
> --
>
> Key: IMPALA-6227
> URL: https://issues.apache.org/jira/browse/IMPALA-6227
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Csaba Ringhofer
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: flaky
> Attachments: TEST-impala-custom-cluster.xml
>
>
> jenkins build https://jenkins.impala.io/job/gerrit-verify-dryrun/1503/console
> failed at the following test:
> {noformat}
> 01:30:11 ] === FAILURES
> ===
> 01:30:11 ] TestAdmissionControllerStress.test_mem_limit[num_queries: 30 |
> submission_delay_ms: 0 | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none | round_robin_submission: True]
> 01:30:11 ] custom_cluster/test_admission_controller.py:877: in test_mem_limit
> 01:30:11 ] {'request_pool': self.pool_name, 'mem_limit': query_mem_limit})
> 01:30:11 ] custom_cluster/test_admission_controller.py:760: in
> run_admission_test
> 01:30:11 ] assert metric_deltas['rejected'] ==\
> 01:30:11 ] E assert 5 == ((30 - 15) - 15)
> {noformat}
> This is probably related to the following recent commit:
> https://github.com/apache/incubator-impala/commit/7487c5de04c2c5d97b8a8d5c935d10568f1ed686
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6227) TestAdmissionControllerStress can be flaky
[
https://issues.apache.org/jira/browse/IMPALA-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464480#comment-16464480
]
Tim Armstrong commented on IMPALA-6227:
---
Attached logs. I think this is the interesting part:
{noformat}
Thread-676: Admitted query 10
-- fetching results from: tests.common.impala_connection.OperationHandle
object at 0x7f090c016250
MainThread: wait_for_metric_changes, initial=admitted=26, queued=20,
dequeued=14, rejected=20, released=18, timed-out=0
MainThread: wait_for_metric_changes, current=admitted=30, queued=20,
dequeued=20, rejected=20, released=24, timed-out=0
MainThread: DeltaSum=6 Deltas={dequeued: 6, admitted:
4, released: 6, rejected: 0, queued: 0,
timed-out: 0} (Expected=6 for metrics=[released])
MainThread: Found all 6 metrics after 0.5 seconds
MainThread: wait_for_metric_changes, initial=admitted=26, queued=20,
dequeued=14, rejected=20, released=18, timed-out=0
MainThread: wait_for_metric_changes, current=admitted=30, queued=20,
dequeued=20, rejected=20, released=24, timed-out=0
MainThread: DeltaSum=4 Deltas={dequeued: 6, admitted:
4, released: 6, rejected: 0, queued: 0,
timed-out: 0} (Expected=5 for metrics=[admitted,
timed-out])
MainThread: wait_for_metric_changes, current=admitted=30, queued=20,
dequeued=20, rejected=20, released=24, timed-out=0
MainThread: DeltaSum=4 Deltas={dequeued: 6, admitted:
4, released: 6, rejected: 0, queued: 0,
timed-out: 0} (Expected=5 for metrics=[admitted,
timed-out])
MainThread: wait_for_metric_changes, current=admitted=30, queued=20,
dequeued=20, rejected=20, released=24, timed-out=0
MainThread: DeltaSum=4 Deltas={dequeued: 6, admitted:
4, released: 6, rejected: 0, queued: 0,
timed-out: 0} (Expected=5 for metrics=[admitted,
timed-out])
MainThread: wait_for_metric_changes, current=admitted=30, queued=20,
dequeued=20, rejected=20, released=24, timed-out=0
MainThread: DeltaSum=4 Deltas={dequeued: 6, admitted:
4, released: 6, rejected: 0, queued: 0,
timed-out: 0} (Expected=5 for metrics=[admitted,
timed-out])
MainThread: wait_for_metric_changes, current=admitted=30, queued=20,
dequeued=20, rejected=20, released=24, timed-out=0
MainThread: DeltaSum=4 Deltas={dequeued: 6, admitted:
4, released: 6, rejected: 0, queued: 0,
timed-out: 0} (Expected=5 for metrics=[admitted,
timed-out])
MainThread: wait_for_metric_changes, current=admitted=30, queued=20,
dequeued=20, rejected=20, released=24, timed-out=0
MainThread: DeltaSum=4 Deltas={dequeued: 6, admitted:
4, released: 6, rejected: 0, queued: 0,
timed-out: 0} (Expected=5 for metrics=[admitted,
timed-out])
{noformat}
> TestAdmissionControllerStress can be flaky
> --
>
> Key: IMPALA-6227
> URL: https://issues.apache.org/jira/browse/IMPALA-6227
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Csaba Ringhofer
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: flaky
> Attachments: TEST-impala-custom-cluster.xml
>
>
> jenkins build https://jenkins.impala.io/job/gerrit-verify-dryrun/1503/console
> failed at the following test:
> {noformat}
> 01:30:11 ] === FAILURES
> ===
> 01:30:11 ] TestAdmissionControllerStress.test_mem_limit[num_queries: 30 |
> submission_delay_ms: 0 | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none | round_robin_submission: True]
> 01:30:11 ] custom_cluster/test_admission_controller.py:877: in test_mem_limit
> 01:30:11 ] {'request_pool': self.pool_name, 'mem_limit': query_mem_limit})
> 01:30:11 ] custom_cluster/test_admission_controller.py:760: in
> run_admission_test
> 01:30:11 ] assert metric_deltas['rejected'] ==\
> 01:30:11 ] E assert 5 == ((30 - 15) - 15)
> {noformat}
> This is probably related to the following recent commit:
> https://github.com/apache/incubator-impala/commit/7487c5de04c2c5d97b8a8d5c935d10568f1ed686
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-6977) Report CPU usage (user/sys) per fragment
Tim Armstrong created IMPALA-6977:
-
Summary: Report CPU usage (user/sys) per fragment
Key: IMPALA-6977
URL: https://issues.apache.org/jira/browse/IMPALA-6977
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Tim Armstrong
The exec summary only shows wallclock time, because that is the only time
reported per operator.
{noformat}
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est.
Peak Mem Detail
---
03:AGGREGATE1 1.782ms 1.782ms 1 1 28.16 KB
10.00 MB FINALIZE
02:EXCHANGE 1 90.073us 90.073us 3 1 104.00 KB
0 UNPARTITIONED
01:AGGREGATE3 1s560ms 1s657ms 3 11.35 MB
10.00 MB
00:SCAN HDFS3 1s567ms 1s815ms 6.00M 6.00M 59.11 MB
144.00 MB tpch_seq_gzip.lineitem
{noformat}
This is fairly useful for understanding query bottlenecks but is misleading whn
it comes to determining where CPU was consumed. We have accurate User and Sys
time for fragment instances so we could have a CPU usage summary based on
profile information
{noformat}
Fragment #Hosts Avg UserTime Max UserTime Avg SysTime Max SysTime
--
F011 12.000ms ... ..
F003 7s245ms 7s624ms ... ..
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6975) TestRuntimeRowFilters.test_row_filters failing with Memory limit exceeded
[
https://issues.apache.org/jira/browse/IMPALA-6975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464269#comment-16464269
]
Tim Armstrong commented on IMPALA-6975:
---
We probably just need to increase mem_limit slightly. Maybe to 250MB?
Here's a more readable mem_tracker dump:
{noformat}
18:08:37 EQuery aborted:Memory limit exceeded:
ParquetColumnReader::ReadDataPage() failed to allocate 65533 bytes for
decompressed data.
18:08:37 E HDFS_SCAN_NODE (id=0) could not allocate 64.00 KB without
exceeding limit.
18:08:37 E Error occurred on backend
impala-boost-static-burst-slave-el7-1aa3.vpc.cloudera.com:22001 by fragment
f9423526590ed30b:732f1db10001
18:08:37 E Memory left in process limit: 11.14 GB
18:08:37 E Memory left in query limit: 22.16 KB
18:08:37 E Query(f9423526590ed30b:732f1db1): Limit=200.00 MB
Reservation=160.00 MB ReservationLimit=160.00 MB OtherMemory=39.98 MB
Total=199.98 MB Peak=199.98 MB
18:08:37 E Fragment f9423526590ed30b:732f1db10007: Reservation=134.00
MB OtherMemory=30.22 MB Total=164.22 MB Peak=164.22 MB
18:08:37 E Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00
MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB
18:08:37 E AGGREGATION_NODE (id=3): Total=4.00 KB Peak=4.00 KB
18:08:37 E Exprs: Total=4.00 KB Peak=4.00 KB
18:08:37 E HASH_JOIN_NODE (id=2): Reservation=132.00 MB
OtherMemory=198.25 KB Total=132.19 MB Peak=132.19 MB
18:08:37 E Exprs: Total=25.12 KB Peak=25.12 KB
18:08:37 E Hash Join Builder (join_node_id=2): Total=157.12 KB
Peak=157.12 KB
18:08:37 E Hash Join Builder (join_node_id=2) Exprs: Total=149.12 KB
Peak=149.12 KB
18:08:37 E EXCHANGE_NODE (id=4): Reservation=14.91 MB OtherMemory=67.76
KB Total=14.97 MB Peak=14.97 MB
18:08:37 E KrpcDeferredRpcs: Total=67.76 KB Peak=67.76 KB
18:08:37 E EXCHANGE_NODE (id=5): Reservation=15.03 MB OtherMemory=0
Total=15.03 MB Peak=15.03 MB
18:08:37 E KrpcDeferredRpcs: Total=0 Peak=45.12 KB
18:08:37 E KrpcDataStreamSender (dst_id=6): Total=16.00 KB Peak=16.00 KB
18:08:37 E Fragment f9423526590ed30b:732f1db10001: Reservation=26.00 MB
OtherMemory=9.75 MB Total=35.75 MB Peak=35.75 MB
18:08:37 E Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00
MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB
18:08:37 E HDFS_SCAN_NODE (id=0): Reservation=24.00 MB OtherMemory=9.30
MB Total=33.30 MB Peak=33.30 MB
18:08:37 E Exprs: Total=260.00 KB Peak=260.00 KB
18:08:37 E KrpcDataStreamSender (dst_id=4): Total=426.57 KB Peak=458.57 KB
18:08:37 E KrpcDataStreamSender (dst_id=4) Exprs: Total=256.00 KB
Peak=256.00 KB
18:08:37 E Fragment f9423526590ed30b:732f1db10004: Reservation=0
OtherMemory=0 Total=0 Peak=29.59 MB
18:08:37 E HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0
Peak=29.32 MB
18:08:37 E KrpcDataStreamSender (dst_id=5): Total=0 Peak=266.57 KB
{noformat}
> TestRuntimeRowFilters.test_row_filters failing with Memory limit exceeded
> -
>
> Key: IMPALA-6975
> URL: https://issues.apache.org/jira/browse/IMPALA-6975
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.0
>Reporter: Sailesh Mukil
>Priority: Critical
> Labels: build-failure
>
> {code:java}
> Error Message
> query_test/test_runtime_filters.py:171: in test_row_filters
> test_file_vars={'$RUNTIME_FILTER_WAIT_TIME_MS' : str(WAIT_TIME_MS)})
> common/impala_test_suite.py:405: in run_test_case result =
> self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:620: in __execute_query return
> impalad_client.execute(query, user=user) common/impala_connection.py:160: in
> execute return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute handle =
> self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:341: in __execute_query
> self.wait_for_completion(handle) beeswax/impala_beeswax.py:361: in
> wait_for_completion raise ImpalaBeeswaxException("Query aborted:" +
> error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E
> Query aborted:Memory limit exceeded: ParquetColumnReader::ReadDataPage()
> failed to allocate 65533 bytes for decompressed data. E HDFS_SCAN_NODE
> (id=0) could not allocate 64.00 KB without exceeding limit. E Error
> occurred on backend
> impala-boost-static-burst-slave-el7-1aa3.vpc.cloudera.com:22001 by fragment
> f9423526590ed30b:732f1db10001 E Memory left in process limit: 11.14 GB
> E Memory left in query limit: 22.16 KB E
> Query(f9423526590ed30b:732f1db1): Limit=200.00 MB
[jira] [Commented] (IMPALA-6970) DiskMgr::AllocateBuffersForRange crashes on failed DCHECK
[
https://issues.apache.org/jira/browse/IMPALA-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464283#comment-16464283
]
Tim Armstrong commented on IMPALA-6970:
---
I can't reproduce locally but there is definitely a race that can happen if a
thread returns its reservation at the same time as another is increasing it.
> DiskMgr::AllocateBuffersForRange crashes on failed DCHECK
> -
>
> Key: IMPALA-6970
> URL: https://issues.apache.org/jira/browse/IMPALA-6970
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.13.0
>Reporter: Sailesh Mukil
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: crash
> Attachments: stacks.txt
>
>
> Similar to IMPALA-6587, but the DCHECK failed in a slightly different way.
> Cannot tell if the root cause is the same as that though without further
> investigation.
> {code:java}
> FSF0503 09:30:26.715791 30750 reservation-tracker.cc:376] Check failed: bytes
> <= unused_reservation() (8388608 vs. 6291456)
> *** Check failure stack trace: ***
> @ 0x4277c1d google::LogMessage::Fail()
> @ 0x42794c2 google::LogMessage::SendToLog()
> @ 0x42775f7 google::LogMessage::Flush()
> @ 0x427abbe google::LogMessageFatal::~LogMessageFatal()
> @ 0x1ef1343 impala::ReservationTracker::AllocateFromLocked()
> @ 0x1ef111d impala::ReservationTracker::AllocateFrom()
> @ 0x1ee8c57
> impala::BufferPool::Client::PrepareToAllocateBuffer()
> @ 0x1ee5543 impala::BufferPool::AllocateBuffer()
> @ 0x2f50f68 impala::io::DiskIoMgr::AllocateBuffersForRange()
> @ 0x1f74762 impala::HdfsScanNodeBase::StartNextScanRange()
> @ 0x1f6b052 impala::HdfsScanNode::ScannerThread()
> @ 0x1f6a4ea
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x1f6c5cc
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x1bd4748 boost::function0<>::operator()()
> @ 0x1ebf349 impala::Thread::SuperviseThread()
> @ 0x1ec74e5 boost::_bi::list5<>::operator()<>()
> @ 0x1ec7409 boost::_bi::bind_t<>::operator()()
> @ 0x1ec73cc boost::detail::thread_data<>::run()
> @ 0x31a1f0a thread_proxy
> @ 0x36d1607851 (unknown)
> @ 0x36d12e894d (unknown)
> {code}
> Git hash of Impala used in job: ba84ad03cb83d7f7aed8524fcfbb0e2cdc9fdd53
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6997) Query execution should notice UDF MemLimitExceeded errors more quickly
[ https://issues.apache.org/jira/browse/IMPALA-6997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468065#comment-16468065 ] Tim Armstrong commented on IMPALA-6997: --- IMPALA-2399 is meant to be the ultimate fix for this. It includes getting rid of SetMemLimitExceeded() entirely but depends on changing the UDF interface significantly. 2. seems like a reasonable mitigation > Query execution should notice UDF MemLimitExceeded errors more quickly > -- > > Key: IMPALA-6997 > URL: https://issues.apache.org/jira/browse/IMPALA-6997 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.13.0 >Reporter: Joe McDonnell >Priority: Major > > When a UDF hits a memory limit, it calls RuntimeState::SetMemLimitExceeded() > which sets the query status, but it has no way of returning status directly. > It relies on the caller checking status periodically. > HdfsTableSink::Send() checks for errors by calling > RuntimeState::CheckQueryState() once at the beginning. If it is evaluating a > UDF and that UDF hits the memory limit, it will need to process the whole > RowBatch before it aborts the query. This could be 1024 rows and each row may > hit a memory limit in that UDF. Other locations that process UDFs may be > processing considerably more rows. > There are two general approaches: > # Code locations should check for status more frequently and thus abort > faster after a RuntimeState::SetMemLImitExceeded() call. > # RuntimeState::SetMemLimitExceeded() should be substantially cheaper, > allowing the rows to be processed faster. > RuntimeState::SetMemLimitExceeded() currently calls > MemTracker::MemLimitExceeded() unconditionally. It then checks to see if it > should update query_status_ (i.e. query_status_ is currently ok). Then it > logs this error. This is wasteful, because MemTracker::MemLimitExceeded() is > not a cheap function, and this is flooding the log for each row. > RuntimeState::SetMemLimitExceeded() should check status before running > MemTracker::MemoryLimitExceeded(). If query_status_ is already not ok, it can > avoid the cost of the dump and logging. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-6995) False-positive DCHECK when converting whitespace-only strings to timestamp
Tim Armstrong created IMPALA-6995:
-
Summary: False-positive DCHECK when converting whitespace-only
strings to timestamp
Key: IMPALA-6995
URL: https://issues.apache.org/jira/browse/IMPALA-6995
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.12.0, Impala 3.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong
{noformat}
select cast(' ' as timestamp);
{noformat}
{noformat}
F0508 14:32:07.245255 11824 timestamp-parse-util.cc:241] Check failed:
dt_ctx->fmt_len > 0 (0 vs. 0)
*** Check failure stack trace: ***
@ 0x428956d google::LogMessage::Fail()
@ 0x428ae12 google::LogMessage::SendToLog()
@ 0x4288f47 google::LogMessage::Flush()
@ 0x428c50e google::LogMessageFatal::~LogMessageFatal()
@ 0x1c3f485 impala::TimestampParser::ParseFormatTokensByStr()
@ 0x1c40553 impala::TimestampParser::Parse()
@ 0x1c4712a impala::TimestampValue::Parse()
@ 0x2e5d8fa impala::CastFunctions::CastToTimestampVal()
@ 0x2e45322 impala::ScalarFnCall::InterpretEval<>()
@ 0x2e27de5 impala::ScalarFnCall::GetTimestampVal()
@ 0x2de72de impala::ScalarExprEvaluator::GetValue()
@ 0x2de6e69 impala::ScalarExprEvaluator::GetValue()
@ 0x1d1dbbf
Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow
@ 0x7fb7cc1d07e8 (unknown)
Picked up JAVA_TOOL_OPTIONS:
-agentlib:jdwp=transport=dt_socket,address=3,server=y,suspend=n
Wrote minidump to
/home/tarmstrong/Impala/incubator-impala/logs/cluster/minidumps/impalad/42afc7f9-5b4a-4ed7-b34ad782-d7904747.dmp
{noformat}
It seems to work fine on a release build.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6996) PartitionedAggregationNode::Close() should not allocate memory
[
https://issues.apache.org/jira/browse/IMPALA-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467999#comment-16467999
]
Tim Armstrong commented on IMPALA-6996:
---
We can't necessarily avoid the memory allocation in general because the UDAF
lifecycle requires that Serialize or Finalize() is called for each intermediate
value. We might be able to avoid it for builtins.
> PartitionedAggregationNode::Close() should not allocate memory
> --
>
> Key: IMPALA-6996
> URL: https://issues.apache.org/jira/browse/IMPALA-6996
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.12.0
>Reporter: Zoram Thanga
>Priority: Critical
>
> A CTAS query with MEM_LIMIT=6GB hit the limit while in an AggregationNode.
> Upon processing the failure, the fragment instance that hit the MEM_LIMIT
> tries to allocate more memory, and repeatedly dumps the following error
> message and stack trace to Impalad log file, once per row.
> {noformat}
> I0508 01:32:10.611563 40614 status.cc:55] Memory limit exceeded:
> FunctionContextImpl::AllocateLocal's allocations exceeded memory limits.
> Exprs could not allocate 3.00 B without exceeding limit.
> Error occurred on backend .com:22000 by fragment
> b429287bdabe184:9e7b5e96002c
> Memory left in process limit: 384.47 GB
> Memory left in query limit: -6947009.00 B
> Query(b429287bdabe184:9e7b5e96): memory limit exceeded. Limit=6.00 GB
> Total=6.01 GB Peak=6.01 GB
> Fragment b429287bdabe184:9e7b5e96002c: Total=2.76 GB Peak=2.76 GB
> AGGREGATION_NODE (id=3): Total=2.75 GB Peak=2.75 GB
> Exprs: Total=751.94 MB Peak=751.94 MB
> EXCHANGE_NODE (id=2): Total=0 Peak=0
> DataStreamRecvr: Total=7.42 MB Peak=7.42 MB
> CodeGen: Total=5.57 KB Peak=745.50 KB
> Block Manager: Limit=4.80 GB Total=4.39 GB Peak=4.40 GB
> Fragment b429287bdabe184:9e7b5e96000c: Total=3.25 GB Peak=3.25 GB
> AGGREGATION_NODE (id=1): Total=3.19 GB Peak=3.19 GB
> Exprs: Total=816.06 MB Peak=816.06 MB
> HDFS_SCAN_NODE (id=0): Total=59.96 MB Peak=200.09 MB
> CodeGen: Total=5.96 KB Peak=876.00 KB
> @ 0x83c78a impala::Status::Status()
> @ 0x83c98e impala::Status::MemLimitExceeded()
> @ 0xa24344 impala::MemTracker::MemLimitExceeded()
> @ 0xa35ccd impala::RuntimeState::SetMemLimitExceeded()
> @ 0xb6bd6d impala::FunctionContextImpl::CheckAllocResult()
> @ 0xb6ae78 impala::FunctionContextImpl::AllocateLocal()
> @ 0xb6b10f impala_udf::StringVal::StringVal()
> @ 0xb6b16a impala_udf::StringVal::CopyFrom()
> @ 0x8a2641 impala::AggregateFunctions::StringValGetValue()
> @ 0x8a2661
> impala::AggregateFunctions::StringValSerializeOrFinalize()
> @ 0xd411c5 impala::AggFnEvaluator::SerializeOrFinalize()
> @ 0xce6569 impala::PartitionedAggregationNode::CleanupHashTbl()
> @ 0xce693b
> impala::PartitionedAggregationNode::Partition::Close()
> @ 0xce8152
> impala::PartitionedAggregationNode::ClosePartitions()
> @ 0xcf163e impala::PartitionedAggregationNode::Close()
> @ 0xa7ae35 impala::FragmentInstanceState::Close()
> @ 0xa7ebfb impala::FragmentInstanceState::Exec()
> @ 0xa6aaf6 impala::QueryState::ExecFInstance()
> @ 0xbef9d9 impala::Thread::SuperviseThread()
> @ 0xbf0394 boost::detail::thread_data<>::run()
> @ 0xe588aa (unknown)
> @ 0x7f8a54c32e25 start_thread
> @ 0x7f8a5496034d __clone
> {noformat}
> Here't the profile summary:
> {noformat}
> Operator #Hosts Avg Time Max Time#Rows Est. #Rows Peak Mem
> Est. Peak Mem Detail
> ---
> 03:AGGREGATE 32 0.000ns 0.000ns0 1.06B2.75 GB
> 165.23 GB FINALIZE
> 02:EXCHANGE32 1s246ms 7s429ms 412.85M 1.06B 0
> 0 HASH()
> 01:AGGREGATE 32 22s539ms 27s631ms 422.16M 1.06B3.09 GB
> 165.23 GB STREAMING
> 00:SCAN HDFS 32 1s221ms 2s093ms 810.12M 1.06B 222.22 MB
> 616.00 MB
> Errors: Memory limit exceeded: Error occurred on backend
> .com:22000 by fragment b429287bdabe184:9e7b5e96002c
> Memory left in process limit: 386.76 GB
> Memory left in query limit: -19080.00 B
> {noformat}
> And the query string:
> {noformat}
> create TABLE TT>
> STORED AS PARQUET
> TBLPROPERTIES('parquet.compression'='SNAPPY')
> T>AS
> T>SELECT
[jira] [Commented] (IMPALA-4268) Allow PlanRootSink to buffer more than a batch of rows
[
https://issues.apache.org/jira/browse/IMPALA-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474669#comment-16474669
]
Tim Armstrong commented on IMPALA-4268:
---
I think it would solve that as a side-effect.
> Allow PlanRootSink to buffer more than a batch of rows
> --
>
> Key: IMPALA-4268
> URL: https://issues.apache.org/jira/browse/IMPALA-4268
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.8.0
>Reporter: Henry Robinson
>Priority: Major
> Labels: resource-management
>
> In IMPALA-2905, we are introducing a {{PlanRootSink}} that handles the
> production of output rows at the root of a plan.
> The implementation in IMPALA-2905 has the plan execute in a separate thread
> to the consumer, which calls {{GetNext()}} to retrieve the rows. However, the
> sender thread will block until {{GetNext()}} is called, so that there are no
> complications about memory usage and ownership due to having several batches
> in flight at one time.
> However, this also leads to many context switches, as each {{GetNext()}} call
> yields to the sender to produce the rows. If the sender was to fill a buffer
> asynchronously, the consumer could pull out of that buffer without taking a
> context switch in many cases (and the extra buffering might smooth out any
> performance spikes due to client delays, which currently directly affect plan
> execution).
> The tricky part is managing the mismatch between the size of the row batches
> processed in {{Send()}} and the size of the fetch result asked for by the
> client. The sender materializes output rows in a {{QueryResultSet}} that is
> owned by the coordinator. That is not, currently, a splittable object -
> instead it contains the actual RPC response struct that will hit the wire
> when the RPC completes. As asynchronous sender cannot know the batch size,
> which may change on every fetch call. So the {{GetNext()}} implementation
> would need to be able to split out the {{QueryResultSet}} to match the
> correct fetch size, and handle stitching together other {{QueryResultSets}} -
> without doing extra copies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7025) PlannerTest.testTableSample has wrong mem-reservation
[
https://issues.apache.org/jira/browse/IMPALA-7025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474666#comment-16474666
]
Tim Armstrong commented on IMPALA-7025:
---
Is this on a 2.x or 3.x branch?
> PlannerTest.testTableSample has wrong mem-reservation
> -
>
> Key: IMPALA-7025
> URL: https://issues.apache.org/jira/browse/IMPALA-7025
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
>
> Seen once on master exhaustive:
> {noformat}
> Error Message
> Section PLAN of query:
> select id from functional_parquet.alltypes tablesample system(10)
> repeatable(1234)
> Actual does not match expected result:
> F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=16.00MB mem-reservation=8.00KB
> thread-reservation=2
> ^^^
> PLAN-ROOT SINK
> | mem-estimate=0B mem-reservation=0B thread-reservation=0
> |
> 00:SCAN HDFS [functional_parquet.alltypes]
>partitions=3/24 files=3 size=23.70KB
>stored statistics:
> table: rows=unavailable size=unavailable
> partitions: 0/24 rows=unavailable
> columns: unavailable
>extrapolated-rows=disabled max-scan-range-rows=unavailable
>mem-estimate=16.00MB mem-reservation=8.00KB thread-reservation=1
>tuple-ids=0 row-size=4B cardinality=unavailable
> Expected:
> F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=16.00MB mem-reservation=16.00KB
> thread-reservation=2
> PLAN-ROOT SINK
> | mem-estimate=0B mem-reservation=0B thread-reservation=0
> |
> 00:SCAN HDFS [functional_parquet.alltypes]
>partitions=3/24 files=3 size=24.23KB
>stored statistics:
> table: rows=unavailable size=unavailable
> partitions: 0/24 rows=unavailable
> columns: unavailable
>extrapolated-rows=disabled max-scan-range-rows=unavailable
>mem-estimate=16.00MB mem-reservation=16.00KB thread-reservation=1
>tuple-ids=0 row-size=4B cardinality=unavailable{noformat}
> This succeeded on the next build, so it is flaky and might not recur.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7025) PlannerTest.testTableSample has wrong mem-reservation
[
https://issues.apache.org/jira/browse/IMPALA-7025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-7025:
-
Assignee: Tim Armstrong
> PlannerTest.testTableSample has wrong mem-reservation
> -
>
> Key: IMPALA-7025
> URL: https://issues.apache.org/jira/browse/IMPALA-7025
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
>
> Seen once on master exhaustive:
> {noformat}
> Error Message
> Section PLAN of query:
> select id from functional_parquet.alltypes tablesample system(10)
> repeatable(1234)
> Actual does not match expected result:
> F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=16.00MB mem-reservation=8.00KB
> thread-reservation=2
> ^^^
> PLAN-ROOT SINK
> | mem-estimate=0B mem-reservation=0B thread-reservation=0
> |
> 00:SCAN HDFS [functional_parquet.alltypes]
>partitions=3/24 files=3 size=23.70KB
>stored statistics:
> table: rows=unavailable size=unavailable
> partitions: 0/24 rows=unavailable
> columns: unavailable
>extrapolated-rows=disabled max-scan-range-rows=unavailable
>mem-estimate=16.00MB mem-reservation=8.00KB thread-reservation=1
>tuple-ids=0 row-size=4B cardinality=unavailable
> Expected:
> F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=16.00MB mem-reservation=16.00KB
> thread-reservation=2
> PLAN-ROOT SINK
> | mem-estimate=0B mem-reservation=0B thread-reservation=0
> |
> 00:SCAN HDFS [functional_parquet.alltypes]
>partitions=3/24 files=3 size=24.23KB
>stored statistics:
> table: rows=unavailable size=unavailable
> partitions: 0/24 rows=unavailable
> columns: unavailable
>extrapolated-rows=disabled max-scan-range-rows=unavailable
>mem-estimate=16.00MB mem-reservation=16.00KB thread-reservation=1
>tuple-ids=0 row-size=4B cardinality=unavailable{noformat}
> This succeeded on the next build, so it is flaky and might not recur.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7027) Multiple Cast to Varchar with different limit fails with "AnalysisException: null CAUSED BY: IllegalArgumentException: "
[
https://issues.apache.org/jira/browse/IMPALA-7027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7027:
--
Target Version: Impala 2.13.0, Impala 3.1.0
> Multiple Cast to Varchar with different limit fails with "AnalysisException:
> null CAUSED BY: IllegalArgumentException: "
>
>
> Key: IMPALA-7027
> URL: https://issues.apache.org/jira/browse/IMPALA-7027
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.9.0
>Reporter: Meenakshi
>Priority: Critical
>
> If we have multiple cast of '' to varchar statements in a impala query which
> has a distinct like below, the query breaks for scenario when the cast to
> varchar limit in the SQL is lower than the previous cast.
>
> Query 1> Fails with " AnalysisException: null CAUSED BY:
> IllegalArgumentException: targetType=VARCHAR(100) type=VARCHAR(101)"
> SELECT DISTINCT CAST('' as VARCHAR(101)) as CLAIM_COMMENTS,CAST('' as
> VARCHAR(100)) as CA_USER_ID FROM pinnga1ph_ds_prd_wh_gbd.concept_flat_dtl
> limit 1
> Where as the below query succeeds
> Query 2> Success
> SELECT DISTINCT CAST('' as VARCHAR(100)) as CLAIM_COMMENTS,CAST('' as
> VARCHAR(101)) as CA_USER_ID FROM pinnga1ph_ds_prd_wh_gbd.concept_flat_dtl
> limit 1
> The Query succeeds is we remove the distinct clause or add set
> enable_expr_rewrites=false;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7027) Multiple Cast to Varchar with different limit fails with "AnalysisException: null CAUSED BY: IllegalArgumentException: "
[
https://issues.apache.org/jira/browse/IMPALA-7027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7027:
--
Priority: Critical (was: Major)
> Multiple Cast to Varchar with different limit fails with "AnalysisException:
> null CAUSED BY: IllegalArgumentException: "
>
>
> Key: IMPALA-7027
> URL: https://issues.apache.org/jira/browse/IMPALA-7027
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.9.0
>Reporter: Meenakshi
>Priority: Critical
>
> If we have multiple cast of '' to varchar statements in a impala query which
> has a distinct like below, the query breaks for scenario when the cast to
> varchar limit in the SQL is lower than the previous cast.
>
> Query 1> Fails with " AnalysisException: null CAUSED BY:
> IllegalArgumentException: targetType=VARCHAR(100) type=VARCHAR(101)"
> SELECT DISTINCT CAST('' as VARCHAR(101)) as CLAIM_COMMENTS,CAST('' as
> VARCHAR(100)) as CA_USER_ID FROM pinnga1ph_ds_prd_wh_gbd.concept_flat_dtl
> limit 1
> Where as the below query succeeds
> Query 2> Success
> SELECT DISTINCT CAST('' as VARCHAR(100)) as CLAIM_COMMENTS,CAST('' as
> VARCHAR(101)) as CA_USER_ID FROM pinnga1ph_ds_prd_wh_gbd.concept_flat_dtl
> limit 1
> The Query succeeds is we remove the distinct clause or add set
> enable_expr_rewrites=false;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7027) Multiple Cast to Varchar with different limit fails with "AnalysisException: null CAUSED BY: IllegalArgumentException: "
[
https://issues.apache.org/jira/browse/IMPALA-7027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7027:
--
Component/s: (was: Clients)
Frontend
> Multiple Cast to Varchar with different limit fails with "AnalysisException:
> null CAUSED BY: IllegalArgumentException: "
>
>
> Key: IMPALA-7027
> URL: https://issues.apache.org/jira/browse/IMPALA-7027
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.9.0
>Reporter: Meenakshi
>Priority: Critical
>
> If we have multiple cast of '' to varchar statements in a impala query which
> has a distinct like below, the query breaks for scenario when the cast to
> varchar limit in the SQL is lower than the previous cast.
>
> Query 1> Fails with " AnalysisException: null CAUSED BY:
> IllegalArgumentException: targetType=VARCHAR(100) type=VARCHAR(101)"
> SELECT DISTINCT CAST('' as VARCHAR(101)) as CLAIM_COMMENTS,CAST('' as
> VARCHAR(100)) as CA_USER_ID FROM pinnga1ph_ds_prd_wh_gbd.concept_flat_dtl
> limit 1
> Where as the below query succeeds
> Query 2> Success
> SELECT DISTINCT CAST('' as VARCHAR(100)) as CLAIM_COMMENTS,CAST('' as
> VARCHAR(101)) as CA_USER_ID FROM pinnga1ph_ds_prd_wh_gbd.concept_flat_dtl
> limit 1
> The Query succeeds is we remove the distinct clause or add set
> enable_expr_rewrites=false;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7022) TestQueries.test_subquery: Subquery must not return more than one row
[
https://issues.apache.org/jira/browse/IMPALA-7022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-7022:
-
Assignee: Zoltán Borók-Nagy
> TestQueries.test_subquery: Subquery must not return more than one row
> -
>
> Key: IMPALA-7022
> URL: https://issues.apache.org/jira/browse/IMPALA-7022
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Zoltán Borók-Nagy
>Priority: Blocker
> Labels: broken-build
>
> Seen on master and on 2.x based branches on exhaustive tests:
> {noformat}
> Error Message
> query_test/test_queries.py:102: in test_subquery
> self.run_test_case('QueryTest/subquery', vector)
> common/impala_test_suite.py:408: in run_test_case
> self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> common/impala_test_suite.py:286: in __verify_exceptions (expected_str,
> actual_str) E AssertionError: Unexpected exception string. Expected: Query
> aborted:Subquery must not return more than one row: SELECT bigint_col FROM
> functional.alltypes_view E Not found in actual: ImpalaBeeswaxException:
> INNER EXCEPTION: MESSAGE: Subquery
> must not return more than one row: SELECT bigint_col FROM
> functional.alltypes_view
> Stacktrace
> query_test/test_queries.py:102: in test_subquery
> self.run_test_case('QueryTest/subquery', vector)
> common/impala_test_suite.py:408: in run_test_case
> self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> common/impala_test_suite.py:286: in __verify_exceptions
> (expected_str, actual_str)
> E AssertionError: Unexpected exception string. Expected: Query
> aborted:Subquery must not return more than one row: SELECT bigint_col FROM
> functional.alltypes_view
> E Not found in actual: ImpalaBeeswaxException: INNER EXCEPTION: 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: Subquery must not return more
> than one row: SELECT bigint_col FROM functional.alltypes_view{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7025) PlannerTest.testTableSample has wrong mem-reservation
[
https://issues.apache.org/jira/browse/IMPALA-7025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475099#comment-16475099
]
Tim Armstrong commented on IMPALA-7025:
---
It could be because a different set of files was selected or because the files
were slightly different sizes. Need to investigate to determine which, although
maybe the ultimate fix is to allow ignoring the resource estimates in these
tests
> PlannerTest.testTableSample has wrong mem-reservation
> -
>
> Key: IMPALA-7025
> URL: https://issues.apache.org/jira/browse/IMPALA-7025
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
>
> Seen once on master exhaustive:
> {noformat}
> Error Message
> Section PLAN of query:
> select id from functional_parquet.alltypes tablesample system(10)
> repeatable(1234)
> Actual does not match expected result:
> F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=16.00MB mem-reservation=8.00KB
> thread-reservation=2
> ^^^
> PLAN-ROOT SINK
> | mem-estimate=0B mem-reservation=0B thread-reservation=0
> |
> 00:SCAN HDFS [functional_parquet.alltypes]
>partitions=3/24 files=3 size=23.70KB
>stored statistics:
> table: rows=unavailable size=unavailable
> partitions: 0/24 rows=unavailable
> columns: unavailable
>extrapolated-rows=disabled max-scan-range-rows=unavailable
>mem-estimate=16.00MB mem-reservation=8.00KB thread-reservation=1
>tuple-ids=0 row-size=4B cardinality=unavailable
> Expected:
> F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=16.00MB mem-reservation=16.00KB
> thread-reservation=2
> PLAN-ROOT SINK
> | mem-estimate=0B mem-reservation=0B thread-reservation=0
> |
> 00:SCAN HDFS [functional_parquet.alltypes]
>partitions=3/24 files=3 size=24.23KB
>stored statistics:
> table: rows=unavailable size=unavailable
> partitions: 0/24 rows=unavailable
> columns: unavailable
>extrapolated-rows=disabled max-scan-range-rows=unavailable
>mem-estimate=16.00MB mem-reservation=16.00KB thread-reservation=1
>tuple-ids=0 row-size=4B cardinality=unavailable{noformat}
> This succeeded on the next build, so it is flaky and might not recur.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-6946) Hit DCHECK in impala::RleBatchDecoder::GetRepeatedValue
[
https://issues.apache.org/jira/browse/IMPALA-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-6946.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
Impala 2.13.0
> Hit DCHECK in impala::RleBatchDecoder::GetRepeatedValue
> -
>
> Key: IMPALA-6946
> URL: https://issues.apache.org/jira/browse/IMPALA-6946
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0, Impala 3.0, Impala 2.12.0, Impala 2.13.0,
> Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: crash
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> The bug comes from conversion between signed and unsigned.
> {noformat}
> DCHECK_GT(num_repeats_to_consume, 0);
> (gdb) p num_repeats_to_consume
> $2 = -1003251240
> {noformat}
> {noformat}
> #2 0x01f422f0 in impala::ScalarColumnReader (parquet::Type::type)6, true>::ReadNonRepeatedValueBatch (this=0x6f547d50,
> pool=0x7fb37bdbb330, max_values=0,
> tuple_size=90, tuple_mem=0x0, num_values=0x7fb37bdbb330) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:252
> #3 0x03fceea7 in google::LogMessage::Flush() ()
> #4 0x03fd246e in google::LogMessageFatal::~LogMessageFatal() ()
> #5 0x01f97197 in impala::RleBatchDecoder int>::GetRepeatedValue (this=0x1803fc6a0, num_repeats_to_consume=-1003251240)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/rle-encoding.h:493
> #6 0x01f96d7b in
> impala::DictDecoder::DecodeNextValue (this=0x1803fc698,
> value=0x16b7e9000)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/dict-encoding.h:455
> #7 0x01f94528 in
> impala::DictDecoder::GetNextValue (value=0x16b7e9000,
> this=0x1803fc698)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/dict-encoding.h:446
> #8 impala::ScalarColumnReader true>::ReadSlot (pool=0x6f547d98, tuple=0x16b7e9000,
> this=0x1803fc400)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:499
> #9 impala::ScalarColumnReader true>::MaterializeValueBatch (this=0x1803fc400,
> pool=0x6f547d98, max_values=1, tuple_size=90,
> tuple_mem=0x16b7e9000 "", num_values=0x7fb37bdbb5f0) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:413
> #10 0x01f67b4a in impala::ScalarColumnReader (parquet::Type::type)6, true>::MaterializeValueBatch
> (this=0x1803fc400, pool=0x6f547d98, max_values=1,
> tuple_size=90, tuple_mem=0x16b7e9000 "", num_values=0x7fb37bdbb5f0) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:436
> #11 0x01f657c2 in impala::ScalarColumnReader (parquet::Type::type)6, true>::ReadValueBatch (this=0x1803fc400,
> pool=0x6f547d98, max_values=1, tuple_size=90,
> tuple_mem=0x16b7e9000 "", num_values=0x6f547d50) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:354
> #12 0x01f422f0 in impala::ScalarColumnReader (parquet::Type::type)6, true>::ReadNonRepeatedValueBatch (this=0x1803fc400,
> pool=0x6f547d98, max_values=1,
> tuple_size=90, tuple_mem=0x16b7e9000 "", num_values=0x6f547d50) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:252
> #13 0x01d70a1c in impala::HdfsParquetScanner::AssembleRows
> (this=0x1a926000, column_readers=..., row_batch=0xdfc0780,
> skip_row_group=0x1a9261b8)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:936
> #14 0x01d6d4f7 in impala::HdfsParquetScanner::GetNextInternal
> (this=0x1a926000, row_batch=0xdfc0780)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:434
> #15 0x01d6b6c6 in impala::HdfsParquetScanner::ProcessSplit
> (this=0x1a926000) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:332
> #16 0x01cf30e4 in impala::HdfsScanNode::ProcessSplit
> (this=0x10c21000, filter_ctxs=..., expr_results_pool=0x7fb37bdbc480,
> scan_range=0xf3276c0, scanner_thread_reservation=0x7fb37bdbc400)
> at
>
[jira] [Assigned] (IMPALA-6035) Add query option that rejects queries based on query complexity
[ https://issues.apache.org/jira/browse/IMPALA-6035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong reassigned IMPALA-6035: - Assignee: Tim Armstrong > Add query option that rejects queries based on query complexity > --- > > Key: IMPALA-6035 > URL: https://issues.apache.org/jira/browse/IMPALA-6035 > Project: IMPALA > Issue Type: Sub-task > Components: Distributed Exec >Reporter: Mostafa Mokhtar >Assignee: Tim Armstrong >Priority: Major > Labels: admission-control > > Expose query option to reject queries with a large number of remote > fragments. > Queries with a large number of fragments can negatively affect cluster > behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6034) Add query option that rejects queries based on estimate number of scanned bytes
[ https://issues.apache.org/jira/browse/IMPALA-6034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong reassigned IMPALA-6034: - Assignee: Mostafa Mokhtar > Add query option that rejects queries based on estimate number of scanned > bytes > > > Key: IMPALA-6034 > URL: https://issues.apache.org/jira/browse/IMPALA-6034 > Project: IMPALA > Issue Type: Sub-task > Components: Distributed Exec >Reporter: Mostafa Mokhtar >Assignee: Mostafa Mokhtar >Priority: Major > > Reject queries that scans large data before executing the query. > This is a mechanism to protect the cluster from potentially harmful queries. > MAX_READ_BYTES: [0] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6996) PartitionedAggregationNode::Close() should not dump stack trace
[
https://issues.apache.org/jira/browse/IMPALA-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469089#comment-16469089
]
Tim Armstrong commented on IMPALA-6996:
---
+1, just avoiding unnecessary work seems like a clear win
> PartitionedAggregationNode::Close() should not dump stack trace
> ---
>
> Key: IMPALA-6996
> URL: https://issues.apache.org/jira/browse/IMPALA-6996
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.12.0
>Reporter: Zoram Thanga
>Priority: Critical
>
> A CTAS query with MEM_LIMIT=6GB hit the limit while in an AggregationNode.
> Upon processing the failure, the fragment instance that hit the MEM_LIMIT
> tries to allocate more memory, and repeatedly dumps the following error
> message and stack trace to Impalad log file, once per row.
> {noformat}
> I0508 01:32:10.611563 40614 status.cc:55] Memory limit exceeded:
> FunctionContextImpl::AllocateLocal's allocations exceeded memory limits.
> Exprs could not allocate 3.00 B without exceeding limit.
> Error occurred on backend .com:22000 by fragment
> b429287bdabe184:9e7b5e96002c
> Memory left in process limit: 384.47 GB
> Memory left in query limit: -6947009.00 B
> Query(b429287bdabe184:9e7b5e96): memory limit exceeded. Limit=6.00 GB
> Total=6.01 GB Peak=6.01 GB
> Fragment b429287bdabe184:9e7b5e96002c: Total=2.76 GB Peak=2.76 GB
> AGGREGATION_NODE (id=3): Total=2.75 GB Peak=2.75 GB
> Exprs: Total=751.94 MB Peak=751.94 MB
> EXCHANGE_NODE (id=2): Total=0 Peak=0
> DataStreamRecvr: Total=7.42 MB Peak=7.42 MB
> CodeGen: Total=5.57 KB Peak=745.50 KB
> Block Manager: Limit=4.80 GB Total=4.39 GB Peak=4.40 GB
> Fragment b429287bdabe184:9e7b5e96000c: Total=3.25 GB Peak=3.25 GB
> AGGREGATION_NODE (id=1): Total=3.19 GB Peak=3.19 GB
> Exprs: Total=816.06 MB Peak=816.06 MB
> HDFS_SCAN_NODE (id=0): Total=59.96 MB Peak=200.09 MB
> CodeGen: Total=5.96 KB Peak=876.00 KB
> @ 0x83c78a impala::Status::Status()
> @ 0x83c98e impala::Status::MemLimitExceeded()
> @ 0xa24344 impala::MemTracker::MemLimitExceeded()
> @ 0xa35ccd impala::RuntimeState::SetMemLimitExceeded()
> @ 0xb6bd6d impala::FunctionContextImpl::CheckAllocResult()
> @ 0xb6ae78 impala::FunctionContextImpl::AllocateLocal()
> @ 0xb6b10f impala_udf::StringVal::StringVal()
> @ 0xb6b16a impala_udf::StringVal::CopyFrom()
> @ 0x8a2641 impala::AggregateFunctions::StringValGetValue()
> @ 0x8a2661
> impala::AggregateFunctions::StringValSerializeOrFinalize()
> @ 0xd411c5 impala::AggFnEvaluator::SerializeOrFinalize()
> @ 0xce6569 impala::PartitionedAggregationNode::CleanupHashTbl()
> @ 0xce693b
> impala::PartitionedAggregationNode::Partition::Close()
> @ 0xce8152
> impala::PartitionedAggregationNode::ClosePartitions()
> @ 0xcf163e impala::PartitionedAggregationNode::Close()
> @ 0xa7ae35 impala::FragmentInstanceState::Close()
> @ 0xa7ebfb impala::FragmentInstanceState::Exec()
> @ 0xa6aaf6 impala::QueryState::ExecFInstance()
> @ 0xbef9d9 impala::Thread::SuperviseThread()
> @ 0xbf0394 boost::detail::thread_data<>::run()
> @ 0xe588aa (unknown)
> @ 0x7f8a54c32e25 start_thread
> @ 0x7f8a5496034d __clone
> {noformat}
> Here't the profile summary:
> {noformat}
> Operator #Hosts Avg Time Max Time#Rows Est. #Rows Peak Mem
> Est. Peak Mem Detail
> ---
> 03:AGGREGATE 32 0.000ns 0.000ns0 1.06B2.75 GB
> 165.23 GB FINALIZE
> 02:EXCHANGE32 1s246ms 7s429ms 412.85M 1.06B 0
> 0 HASH()
> 01:AGGREGATE 32 22s539ms 27s631ms 422.16M 1.06B3.09 GB
> 165.23 GB STREAMING
> 00:SCAN HDFS 32 1s221ms 2s093ms 810.12M 1.06B 222.22 MB
> 616.00 MB
> Errors: Memory limit exceeded: Error occurred on backend
> .com:22000 by fragment b429287bdabe184:9e7b5e96002c
> Memory left in process limit: 386.76 GB
> Memory left in query limit: -19080.00 B
> {noformat}
> And the query string:
> {noformat}
> create TABLE TT>
> STORED AS PARQUET
> TBLPROPERTIES('parquet.compression'='SNAPPY')
> T>AS
> T>SELECT T>,
> ,
> ,
> MAX() AS ,
>
[jira] [Updated] (IMPALA-6034) Add query option that limits scanned bytes at runtime
[ https://issues.apache.org/jira/browse/IMPALA-6034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-6034: -- Summary: Add query option that limits scanned bytes at runtime (was: Add query option that rejects queries based on estimate number of scanned bytes ) > Add query option that limits scanned bytes at runtime > - > > Key: IMPALA-6034 > URL: https://issues.apache.org/jira/browse/IMPALA-6034 > Project: IMPALA > Issue Type: Sub-task > Components: Distributed Exec >Reporter: Mostafa Mokhtar >Assignee: Mostafa Mokhtar >Priority: Major > > Reject queries that scans large data before executing the query. > This is a mechanism to protect the cluster from potentially harmful queries. > MAX_READ_BYTES: [0] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6996) PartitionedAggregationNode::Close() should not dump stack trace
[
https://issues.apache.org/jira/browse/IMPALA-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469237#comment-16469237
]
Tim Armstrong commented on IMPALA-6996:
---
Just to add some context about why this is so slow, since I looked into it. We
use this function from glog:
https://github.com/google/glog/blob/d8cb47f77d1c31779f3ff890e1a5748483778d6a/src/utilities.cc#L119
{code}
static void DumpStackTrace(int skip_count, DebugWriter *writerfn, void *arg) {
// Print stack trace
void* stack[32];
int depth = GetStackTrace(stack, ARRAYSIZE(stack), skip_count+1);
for (int i = 0; i < depth; i++) {
#if defined(HAVE_SYMBOLIZE)
if (FLAGS_symbolize_stacktrace) {
DumpPCAndSymbol(writerfn, arg, stack[i], "");
} else {
DumpPC(writerfn, arg, stack[i], "");
}
#else
DumpPC(writerfn, arg, stack[i], "");
#endif
}
{code}
The expensive part is the symbolisation (which we can disable with a
command-line flag). For every stack frame, the symbolisation function does the
following (based on
https://github.com/google/glog/blob/d8cb47f77d1c31779f3ff890e1a5748483778d6a/src/symbolize.cc#L726):
# Does a linear search through /proc/self/maps for the address
# Opens the object file containing the address
# Reads the ELF header and resolves the address
> PartitionedAggregationNode::Close() should not dump stack trace
> ---
>
> Key: IMPALA-6996
> URL: https://issues.apache.org/jira/browse/IMPALA-6996
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.12.0
>Reporter: Zoram Thanga
>Priority: Critical
>
> A CTAS query with MEM_LIMIT=6GB hit the limit while in an AggregationNode.
> Upon processing the failure, the fragment instance that hit the MEM_LIMIT
> tries to allocate more memory, and repeatedly dumps the following error
> message and stack trace to Impalad log file, once per row.
> {noformat}
> I0508 01:32:10.611563 40614 status.cc:55] Memory limit exceeded:
> FunctionContextImpl::AllocateLocal's allocations exceeded memory limits.
> Exprs could not allocate 3.00 B without exceeding limit.
> Error occurred on backend .com:22000 by fragment
> b429287bdabe184:9e7b5e96002c
> Memory left in process limit: 384.47 GB
> Memory left in query limit: -6947009.00 B
> Query(b429287bdabe184:9e7b5e96): memory limit exceeded. Limit=6.00 GB
> Total=6.01 GB Peak=6.01 GB
> Fragment b429287bdabe184:9e7b5e96002c: Total=2.76 GB Peak=2.76 GB
> AGGREGATION_NODE (id=3): Total=2.75 GB Peak=2.75 GB
> Exprs: Total=751.94 MB Peak=751.94 MB
> EXCHANGE_NODE (id=2): Total=0 Peak=0
> DataStreamRecvr: Total=7.42 MB Peak=7.42 MB
> CodeGen: Total=5.57 KB Peak=745.50 KB
> Block Manager: Limit=4.80 GB Total=4.39 GB Peak=4.40 GB
> Fragment b429287bdabe184:9e7b5e96000c: Total=3.25 GB Peak=3.25 GB
> AGGREGATION_NODE (id=1): Total=3.19 GB Peak=3.19 GB
> Exprs: Total=816.06 MB Peak=816.06 MB
> HDFS_SCAN_NODE (id=0): Total=59.96 MB Peak=200.09 MB
> CodeGen: Total=5.96 KB Peak=876.00 KB
> @ 0x83c78a impala::Status::Status()
> @ 0x83c98e impala::Status::MemLimitExceeded()
> @ 0xa24344 impala::MemTracker::MemLimitExceeded()
> @ 0xa35ccd impala::RuntimeState::SetMemLimitExceeded()
> @ 0xb6bd6d impala::FunctionContextImpl::CheckAllocResult()
> @ 0xb6ae78 impala::FunctionContextImpl::AllocateLocal()
> @ 0xb6b10f impala_udf::StringVal::StringVal()
> @ 0xb6b16a impala_udf::StringVal::CopyFrom()
> @ 0x8a2641 impala::AggregateFunctions::StringValGetValue()
> @ 0x8a2661
> impala::AggregateFunctions::StringValSerializeOrFinalize()
> @ 0xd411c5 impala::AggFnEvaluator::SerializeOrFinalize()
> @ 0xce6569 impala::PartitionedAggregationNode::CleanupHashTbl()
> @ 0xce693b
> impala::PartitionedAggregationNode::Partition::Close()
> @ 0xce8152
> impala::PartitionedAggregationNode::ClosePartitions()
> @ 0xcf163e impala::PartitionedAggregationNode::Close()
> @ 0xa7ae35 impala::FragmentInstanceState::Close()
> @ 0xa7ebfb impala::FragmentInstanceState::Exec()
> @ 0xa6aaf6 impala::QueryState::ExecFInstance()
> @ 0xbef9d9 impala::Thread::SuperviseThread()
> @ 0xbf0394 boost::detail::thread_data<>::run()
> @ 0xe588aa (unknown)
> @ 0x7f8a54c32e25 start_thread
> @ 0x7f8a5496034d __clone
> {noformat}
> Here't the profile summary:
> {noformat}
> Operator #Hosts Avg Time Max Time#Rows Est. #Rows Peak Mem
> Est. Peak Mem Detail
>
[jira] [Resolved] (IMPALA-605) HBASE_CACHING default incorrectly displayed
[
https://issues.apache.org/jira/browse/IMPALA-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-605.
--
Resolution: Duplicate
> HBASE_CACHING default incorrectly displayed
> ---
>
> Key: IMPALA-605
> URL: https://issues.apache.org/jira/browse/IMPALA-605
> Project: IMPALA
> Issue Type: Bug
> Components: Clients
>Affects Versions: Impala 1.1.1
>Reporter: Ricky Saltzer
>Priority: Minor
> Labels: shell
>
> In the impala-shell, the default value of _HBASE_CACHING_ is shown as *0*,
> when in fact it's *1000* (I think).
> {code}
> [my-impala-daemon:21000] > SET;
> Default query options:
> ABORT_ON_DEFAULT_LIMIT_EXCEEDED: 0
> ABORT_ON_ERROR: 0
> ALLOW_UNSUPPORTED_FORMATS: 0
> BATCH_SIZE: 0
> DEBUG_ACTION:
> DEFAULT_ORDER_BY_LIMIT: -1
> DISABLE_CODEGEN: 0
> HBASE_CACHE_BLOCKS: 0
> HBASE_CACHING: 0 <--
> MAX_ERRORS: 0
> MAX_IO_BUFFERS: 0
> MAX_SCAN_RANGE_LENGTH: 0
> MEM_LIMIT: 0
> NUM_NODES: 0
> NUM_SCANNER_THREADS: 0
> PARQUET_COMPRESSION_CODEC: SNAPPY
> SUPPORT_START_OVER: false
> {code}
> This can be pretty confusing since it's implicitly using a different value.
> In my case, it was reading my HBase table which has *very* wide rows, and it
> was causing the region servers to fall over.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-6946) Hit DCHECK in impala::RleBatchDecoder::GetRepeatedValue
[
https://issues.apache.org/jira/browse/IMPALA-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-6946:
--
Target Version: Impala 2.13.0, Impala 3.1.0
> Hit DCHECK in impala::RleBatchDecoder::GetRepeatedValue
> -
>
> Key: IMPALA-6946
> URL: https://issues.apache.org/jira/browse/IMPALA-6946
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0, Impala 3.0, Impala 2.12.0, Impala 2.13.0,
> Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: crash
>
> The bug comes from conversion between signed and unsigned.
> {noformat}
> DCHECK_GT(num_repeats_to_consume, 0);
> (gdb) p num_repeats_to_consume
> $2 = -1003251240
> {noformat}
> {noformat}
> #2 0x01f422f0 in impala::ScalarColumnReader (parquet::Type::type)6, true>::ReadNonRepeatedValueBatch (this=0x6f547d50,
> pool=0x7fb37bdbb330, max_values=0,
> tuple_size=90, tuple_mem=0x0, num_values=0x7fb37bdbb330) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:252
> #3 0x03fceea7 in google::LogMessage::Flush() ()
> #4 0x03fd246e in google::LogMessageFatal::~LogMessageFatal() ()
> #5 0x01f97197 in impala::RleBatchDecoder int>::GetRepeatedValue (this=0x1803fc6a0, num_repeats_to_consume=-1003251240)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/rle-encoding.h:493
> #6 0x01f96d7b in
> impala::DictDecoder::DecodeNextValue (this=0x1803fc698,
> value=0x16b7e9000)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/dict-encoding.h:455
> #7 0x01f94528 in
> impala::DictDecoder::GetNextValue (value=0x16b7e9000,
> this=0x1803fc698)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/dict-encoding.h:446
> #8 impala::ScalarColumnReader true>::ReadSlot (pool=0x6f547d98, tuple=0x16b7e9000,
> this=0x1803fc400)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:499
> #9 impala::ScalarColumnReader true>::MaterializeValueBatch (this=0x1803fc400,
> pool=0x6f547d98, max_values=1, tuple_size=90,
> tuple_mem=0x16b7e9000 "", num_values=0x7fb37bdbb5f0) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:413
> #10 0x01f67b4a in impala::ScalarColumnReader (parquet::Type::type)6, true>::MaterializeValueBatch
> (this=0x1803fc400, pool=0x6f547d98, max_values=1,
> tuple_size=90, tuple_mem=0x16b7e9000 "", num_values=0x7fb37bdbb5f0) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:436
> #11 0x01f657c2 in impala::ScalarColumnReader (parquet::Type::type)6, true>::ReadValueBatch (this=0x1803fc400,
> pool=0x6f547d98, max_values=1, tuple_size=90,
> tuple_mem=0x16b7e9000 "", num_values=0x6f547d50) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:354
> #12 0x01f422f0 in impala::ScalarColumnReader (parquet::Type::type)6, true>::ReadNonRepeatedValueBatch (this=0x1803fc400,
> pool=0x6f547d98, max_values=1,
> tuple_size=90, tuple_mem=0x16b7e9000 "", num_values=0x6f547d50) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:252
> #13 0x01d70a1c in impala::HdfsParquetScanner::AssembleRows
> (this=0x1a926000, column_readers=..., row_batch=0xdfc0780,
> skip_row_group=0x1a9261b8)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:936
> #14 0x01d6d4f7 in impala::HdfsParquetScanner::GetNextInternal
> (this=0x1a926000, row_batch=0xdfc0780)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:434
> #15 0x01d6b6c6 in impala::HdfsParquetScanner::ProcessSplit
> (this=0x1a926000) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:332
> #16 0x01cf30e4 in impala::HdfsScanNode::ProcessSplit
> (this=0x10c21000, filter_ctxs=..., expr_results_pool=0x7fb37bdbc480,
> scan_range=0xf3276c0, scanner_thread_reservation=0x7fb37bdbc400)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:482
> #17
[jira] [Updated] (IMPALA-7010) Multiple flaky tests failing with MemLimitExceeded on S3
[
https://issues.apache.org/jira/browse/IMPALA-7010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7010:
--
Labels: flaky (was: )
> Multiple flaky tests failing with MemLimitExceeded on S3
>
>
> Key: IMPALA-7010
> URL: https://issues.apache.org/jira/browse/IMPALA-7010
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.0, Impala 2.13.0
>Reporter: Sailesh Mukil
>Priority: Blocker
> Labels: flaky
>
> *test_low_mem_limit_orderby_all*
> {code:java}
> Error Message
> query_test/test_mem_usage_scaling.py:272: in test_low_mem_limit_orderby_all
> self.run_primitive_query(vector, 'primitive_orderby_all')
> query_test/test_mem_usage_scaling.py:260: in run_primitive_query
> self.low_memory_limit_test(vector, query_name, self.MIN_MEM[query_name])
> query_test/test_mem_usage_scaling.py:114: in low_memory_limit_test
> self.run_test_case(tpch_query, new_vector) common/impala_test_suite.py:405:
> in run_test_case result = self.__execute_query(target_impalad_client,
> query, user=user) common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute return
> self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute handle =
> self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:341: in __execute_query
> self.wait_for_completion(handle) beeswax/impala_beeswax.py:361: in
> wait_for_completion raise ImpalaBeeswaxException("Query aborted:" +
> error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E
> Query aborted:Memory limit exceeded: Failed to allocate tuple buffer E
> HDFS_SCAN_NODE (id=0) could not allocate 190.00 KB without exceeding limit. E
> Error occurred on backend
> ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment
> db44c56dcd2fce95:7d746e080003 E Memory left in process limit: 11.40 GB
> E Memory left in query limit: 51.61 KB E
> Query(db44c56dcd2fce95:7d746e08): Limit=200.00 MB Reservation=158.50
> MB ReservationLimit=160.00 MB OtherMemory=41.45 MB Total=199.95 MB
> Peak=199.95 MB E Fragment db44c56dcd2fce95:7d746e080003:
> Reservation=158.50 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB E
> SORT_NODE (id=1): Reservation=9.00 MB OtherMemory=8.00 KB Total=9.01 MB
> Peak=22.31 MB E HDFS_SCAN_NODE (id=0): Reservation=149.50 MB
> OtherMemory=41.43 MB Total=190.93 MB Peak=192.13 MB E Exprs:
> Total=4.00 KB Peak=4.00 KB E KrpcDataStreamSender (dst_id=4):
> Total=688.00 B Peak=688.00 B E CodeGen: Total=7.72 KB Peak=973.50 KB E
> E Memory limit exceeded: Failed to allocate tuple buffer E
> HDFS_SCAN_NODE (id=0) could not allocate 190.00 KB without exceeding limit. E
> Error occurred on backend
> ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment
> db44c56dcd2fce95:7d746e080003 E Memory left in process limit: 11.40 GB
> E Memory left in query limit: 51.61 KB E
> Query(db44c56dcd2fce95:7d746e08): Limit=200.00 MB Reservation=158.50
> MB ReservationLimit=160.00 MB OtherMemory=41.45 MB Total=199.95 MB
> Peak=199.95 MB E Fragment db44c56dcd2fce95:7d746e080003:
> Reservation=158.50 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB E
> SORT_NODE (id=1): Reservation=9.00 MB OtherMemory=8.00 KB Total=9.01 MB
> Peak=22.31 MB E HDFS_SCAN_NODE (id=0): Reservation=149.50 MB
> OtherMemory=41.43 MB Total=190.93 MB Peak=192.13 MB E Exprs:
> Total=4.00 KB Peak=4.00 KB E KrpcDataStreamSender (dst_id=4):
> Total=688.00 B Peak=688.00 B E CodeGen: Total=7.72 KB Peak=973.50 KB (1
> of 3 similar)
> Stacktrace
> query_test/test_mem_usage_scaling.py:272: in test_low_mem_limit_orderby_all
> self.run_primitive_query(vector, 'primitive_orderby_all')
> query_test/test_mem_usage_scaling.py:260: in run_primitive_query
> self.low_memory_limit_test(vector, query_name, self.MIN_MEM[query_name])
> query_test/test_mem_usage_scaling.py:114: in low_memory_limit_test
> self.run_test_case(tpch_query, new_vector)
> common/impala_test_suite.py:405: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:341: in __execute_query
>
[jira] [Resolved] (IMPALA-5735) Avoid fork() in ProcessStateInfo
[ https://issues.apache.org/jira/browse/IMPALA-5735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-5735. --- Resolution: Duplicate > Avoid fork() in ProcessStateInfo > > > Key: IMPALA-5735 > URL: https://issues.apache.org/jira/browse/IMPALA-5735 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, > Impala 2.9.0, Impala 2.10.0 >Reporter: Tim Armstrong >Priority: Major > > ReadProcFileDescriptorInfo() forks some processes, which could lead to > unpredictable behaviour or failures when the VM size of the process gets > large. > Instead it could read the file directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7010) Multiple flaky tests failing with MemLimitExceeded on S3
[
https://issues.apache.org/jira/browse/IMPALA-7010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-7010 started by Tim Armstrong.
-
> Multiple flaky tests failing with MemLimitExceeded on S3
>
>
> Key: IMPALA-7010
> URL: https://issues.apache.org/jira/browse/IMPALA-7010
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.0, Impala 2.13.0
>Reporter: Sailesh Mukil
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: flaky
>
> *test_low_mem_limit_orderby_all*
> {code:java}
> Error Message
> query_test/test_mem_usage_scaling.py:272: in test_low_mem_limit_orderby_all
> self.run_primitive_query(vector, 'primitive_orderby_all')
> query_test/test_mem_usage_scaling.py:260: in run_primitive_query
> self.low_memory_limit_test(vector, query_name, self.MIN_MEM[query_name])
> query_test/test_mem_usage_scaling.py:114: in low_memory_limit_test
> self.run_test_case(tpch_query, new_vector) common/impala_test_suite.py:405:
> in run_test_case result = self.__execute_query(target_impalad_client,
> query, user=user) common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute return
> self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute handle =
> self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:341: in __execute_query
> self.wait_for_completion(handle) beeswax/impala_beeswax.py:361: in
> wait_for_completion raise ImpalaBeeswaxException("Query aborted:" +
> error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E
> Query aborted:Memory limit exceeded: Failed to allocate tuple buffer E
> HDFS_SCAN_NODE (id=0) could not allocate 190.00 KB without exceeding limit. E
> Error occurred on backend
> ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment
> db44c56dcd2fce95:7d746e080003 E Memory left in process limit: 11.40 GB
> E Memory left in query limit: 51.61 KB E
> Query(db44c56dcd2fce95:7d746e08): Limit=200.00 MB Reservation=158.50
> MB ReservationLimit=160.00 MB OtherMemory=41.45 MB Total=199.95 MB
> Peak=199.95 MB E Fragment db44c56dcd2fce95:7d746e080003:
> Reservation=158.50 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB E
> SORT_NODE (id=1): Reservation=9.00 MB OtherMemory=8.00 KB Total=9.01 MB
> Peak=22.31 MB E HDFS_SCAN_NODE (id=0): Reservation=149.50 MB
> OtherMemory=41.43 MB Total=190.93 MB Peak=192.13 MB E Exprs:
> Total=4.00 KB Peak=4.00 KB E KrpcDataStreamSender (dst_id=4):
> Total=688.00 B Peak=688.00 B E CodeGen: Total=7.72 KB Peak=973.50 KB E
> E Memory limit exceeded: Failed to allocate tuple buffer E
> HDFS_SCAN_NODE (id=0) could not allocate 190.00 KB without exceeding limit. E
> Error occurred on backend
> ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment
> db44c56dcd2fce95:7d746e080003 E Memory left in process limit: 11.40 GB
> E Memory left in query limit: 51.61 KB E
> Query(db44c56dcd2fce95:7d746e08): Limit=200.00 MB Reservation=158.50
> MB ReservationLimit=160.00 MB OtherMemory=41.45 MB Total=199.95 MB
> Peak=199.95 MB E Fragment db44c56dcd2fce95:7d746e080003:
> Reservation=158.50 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB E
> SORT_NODE (id=1): Reservation=9.00 MB OtherMemory=8.00 KB Total=9.01 MB
> Peak=22.31 MB E HDFS_SCAN_NODE (id=0): Reservation=149.50 MB
> OtherMemory=41.43 MB Total=190.93 MB Peak=192.13 MB E Exprs:
> Total=4.00 KB Peak=4.00 KB E KrpcDataStreamSender (dst_id=4):
> Total=688.00 B Peak=688.00 B E CodeGen: Total=7.72 KB Peak=973.50 KB (1
> of 3 similar)
> Stacktrace
> query_test/test_mem_usage_scaling.py:272: in test_low_mem_limit_orderby_all
> self.run_primitive_query(vector, 'primitive_orderby_all')
> query_test/test_mem_usage_scaling.py:260: in run_primitive_query
> self.low_memory_limit_test(vector, query_name, self.MIN_MEM[query_name])
> query_test/test_mem_usage_scaling.py:114: in low_memory_limit_test
> self.run_test_case(tpch_query, new_vector)
> common/impala_test_suite.py:405: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:341: in
[jira] [Commented] (IMPALA-7010) Multiple flaky tests failing with MemLimitExceeded on S3
[
https://issues.apache.org/jira/browse/IMPALA-7010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471252#comment-16471252
]
Tim Armstrong commented on IMPALA-7010:
---
I'm looking at this. I'll try to move a bunch of these mem_limit tests into
separate tests that we can skip on S3 and other filesystems with different
timing.
> Multiple flaky tests failing with MemLimitExceeded on S3
>
>
> Key: IMPALA-7010
> URL: https://issues.apache.org/jira/browse/IMPALA-7010
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.0, Impala 2.13.0
>Reporter: Sailesh Mukil
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: flaky
>
> *test_low_mem_limit_orderby_all*
> {code:java}
> Error Message
> query_test/test_mem_usage_scaling.py:272: in test_low_mem_limit_orderby_all
> self.run_primitive_query(vector, 'primitive_orderby_all')
> query_test/test_mem_usage_scaling.py:260: in run_primitive_query
> self.low_memory_limit_test(vector, query_name, self.MIN_MEM[query_name])
> query_test/test_mem_usage_scaling.py:114: in low_memory_limit_test
> self.run_test_case(tpch_query, new_vector) common/impala_test_suite.py:405:
> in run_test_case result = self.__execute_query(target_impalad_client,
> query, user=user) common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute return
> self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute handle =
> self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:341: in __execute_query
> self.wait_for_completion(handle) beeswax/impala_beeswax.py:361: in
> wait_for_completion raise ImpalaBeeswaxException("Query aborted:" +
> error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E
> Query aborted:Memory limit exceeded: Failed to allocate tuple buffer E
> HDFS_SCAN_NODE (id=0) could not allocate 190.00 KB without exceeding limit. E
> Error occurred on backend
> ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment
> db44c56dcd2fce95:7d746e080003 E Memory left in process limit: 11.40 GB
> E Memory left in query limit: 51.61 KB E
> Query(db44c56dcd2fce95:7d746e08): Limit=200.00 MB Reservation=158.50
> MB ReservationLimit=160.00 MB OtherMemory=41.45 MB Total=199.95 MB
> Peak=199.95 MB E Fragment db44c56dcd2fce95:7d746e080003:
> Reservation=158.50 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB E
> SORT_NODE (id=1): Reservation=9.00 MB OtherMemory=8.00 KB Total=9.01 MB
> Peak=22.31 MB E HDFS_SCAN_NODE (id=0): Reservation=149.50 MB
> OtherMemory=41.43 MB Total=190.93 MB Peak=192.13 MB E Exprs:
> Total=4.00 KB Peak=4.00 KB E KrpcDataStreamSender (dst_id=4):
> Total=688.00 B Peak=688.00 B E CodeGen: Total=7.72 KB Peak=973.50 KB E
> E Memory limit exceeded: Failed to allocate tuple buffer E
> HDFS_SCAN_NODE (id=0) could not allocate 190.00 KB without exceeding limit. E
> Error occurred on backend
> ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment
> db44c56dcd2fce95:7d746e080003 E Memory left in process limit: 11.40 GB
> E Memory left in query limit: 51.61 KB E
> Query(db44c56dcd2fce95:7d746e08): Limit=200.00 MB Reservation=158.50
> MB ReservationLimit=160.00 MB OtherMemory=41.45 MB Total=199.95 MB
> Peak=199.95 MB E Fragment db44c56dcd2fce95:7d746e080003:
> Reservation=158.50 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB E
> SORT_NODE (id=1): Reservation=9.00 MB OtherMemory=8.00 KB Total=9.01 MB
> Peak=22.31 MB E HDFS_SCAN_NODE (id=0): Reservation=149.50 MB
> OtherMemory=41.43 MB Total=190.93 MB Peak=192.13 MB E Exprs:
> Total=4.00 KB Peak=4.00 KB E KrpcDataStreamSender (dst_id=4):
> Total=688.00 B Peak=688.00 B E CodeGen: Total=7.72 KB Peak=973.50 KB (1
> of 3 similar)
> Stacktrace
> query_test/test_mem_usage_scaling.py:272: in test_low_mem_limit_orderby_all
> self.run_primitive_query(vector, 'primitive_orderby_all')
> query_test/test_mem_usage_scaling.py:260: in run_primitive_query
> self.low_memory_limit_test(vector, query_name, self.MIN_MEM[query_name])
> query_test/test_mem_usage_scaling.py:114: in low_memory_limit_test
> self.run_test_case(tpch_query, new_vector)
> common/impala_test_suite.py:405: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return
[jira] [Resolved] (IMPALA-7010) Multiple flaky tests failing with MemLimitExceeded on S3
[
https://issues.apache.org/jira/browse/IMPALA-7010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-7010.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
Impala 2.13.0
> Multiple flaky tests failing with MemLimitExceeded on S3
>
>
> Key: IMPALA-7010
> URL: https://issues.apache.org/jira/browse/IMPALA-7010
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.0, Impala 2.13.0
>Reporter: Sailesh Mukil
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: flaky
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> *test_low_mem_limit_orderby_all*
> {code:java}
> Error Message
> query_test/test_mem_usage_scaling.py:272: in test_low_mem_limit_orderby_all
> self.run_primitive_query(vector, 'primitive_orderby_all')
> query_test/test_mem_usage_scaling.py:260: in run_primitive_query
> self.low_memory_limit_test(vector, query_name, self.MIN_MEM[query_name])
> query_test/test_mem_usage_scaling.py:114: in low_memory_limit_test
> self.run_test_case(tpch_query, new_vector) common/impala_test_suite.py:405:
> in run_test_case result = self.__execute_query(target_impalad_client,
> query, user=user) common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute return
> self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute handle =
> self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:341: in __execute_query
> self.wait_for_completion(handle) beeswax/impala_beeswax.py:361: in
> wait_for_completion raise ImpalaBeeswaxException("Query aborted:" +
> error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E
> Query aborted:Memory limit exceeded: Failed to allocate tuple buffer E
> HDFS_SCAN_NODE (id=0) could not allocate 190.00 KB without exceeding limit. E
> Error occurred on backend
> ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment
> db44c56dcd2fce95:7d746e080003 E Memory left in process limit: 11.40 GB
> E Memory left in query limit: 51.61 KB E
> Query(db44c56dcd2fce95:7d746e08): Limit=200.00 MB Reservation=158.50
> MB ReservationLimit=160.00 MB OtherMemory=41.45 MB Total=199.95 MB
> Peak=199.95 MB E Fragment db44c56dcd2fce95:7d746e080003:
> Reservation=158.50 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB E
> SORT_NODE (id=1): Reservation=9.00 MB OtherMemory=8.00 KB Total=9.01 MB
> Peak=22.31 MB E HDFS_SCAN_NODE (id=0): Reservation=149.50 MB
> OtherMemory=41.43 MB Total=190.93 MB Peak=192.13 MB E Exprs:
> Total=4.00 KB Peak=4.00 KB E KrpcDataStreamSender (dst_id=4):
> Total=688.00 B Peak=688.00 B E CodeGen: Total=7.72 KB Peak=973.50 KB E
> E Memory limit exceeded: Failed to allocate tuple buffer E
> HDFS_SCAN_NODE (id=0) could not allocate 190.00 KB without exceeding limit. E
> Error occurred on backend
> ec2-m2-4xlarge-centos-6-4-0e8b.vpc.cloudera.com:22001 by fragment
> db44c56dcd2fce95:7d746e080003 E Memory left in process limit: 11.40 GB
> E Memory left in query limit: 51.61 KB E
> Query(db44c56dcd2fce95:7d746e08): Limit=200.00 MB Reservation=158.50
> MB ReservationLimit=160.00 MB OtherMemory=41.45 MB Total=199.95 MB
> Peak=199.95 MB E Fragment db44c56dcd2fce95:7d746e080003:
> Reservation=158.50 MB OtherMemory=41.45 MB Total=199.95 MB Peak=199.95 MB E
> SORT_NODE (id=1): Reservation=9.00 MB OtherMemory=8.00 KB Total=9.01 MB
> Peak=22.31 MB E HDFS_SCAN_NODE (id=0): Reservation=149.50 MB
> OtherMemory=41.43 MB Total=190.93 MB Peak=192.13 MB E Exprs:
> Total=4.00 KB Peak=4.00 KB E KrpcDataStreamSender (dst_id=4):
> Total=688.00 B Peak=688.00 B E CodeGen: Total=7.72 KB Peak=973.50 KB (1
> of 3 similar)
> Stacktrace
> query_test/test_mem_usage_scaling.py:272: in test_low_mem_limit_orderby_all
> self.run_primitive_query(vector, 'primitive_orderby_all')
> query_test/test_mem_usage_scaling.py:260: in run_primitive_query
> self.low_memory_limit_test(vector, query_name, self.MIN_MEM[query_name])
> query_test/test_mem_usage_scaling.py:114: in low_memory_limit_test
> self.run_test_case(tpch_query, new_vector)
> common/impala_test_suite.py:405: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in
[jira] [Commented] (IMPALA-4268) Allow PlanRootSink to buffer more than a batch of rows
[
https://issues.apache.org/jira/browse/IMPALA-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472642#comment-16472642
]
Tim Armstrong commented on IMPALA-4268:
---
I'm going to steal this JIRA and extend it slightly. I think we should do this
in a way that execution resources can be released without the client fetching
all the resources. I.e. if the result set is reasonably small, we should buffer
all the results in the ClientRequestState and release all of the query
execution resources. This would make Impala's resource consumption less hostage
to the behaviour of clients.
> Allow PlanRootSink to buffer more than a batch of rows
> --
>
> Key: IMPALA-4268
> URL: https://issues.apache.org/jira/browse/IMPALA-4268
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.8.0
>Reporter: Henry Robinson
>Priority: Major
> Labels: resource-management
>
> In IMPALA-2905, we are introducing a {{PlanRootSink}} that handles the
> production of output rows at the root of a plan.
> The implementation in IMPALA-2905 has the plan execute in a separate thread
> to the consumer, which calls {{GetNext()}} to retrieve the rows. However, the
> sender thread will block until {{GetNext()}} is called, so that there are no
> complications about memory usage and ownership due to having several batches
> in flight at one time.
> However, this also leads to many context switches, as each {{GetNext()}} call
> yields to the sender to produce the rows. If the sender was to fill a buffer
> asynchronously, the consumer could pull out of that buffer without taking a
> context switch in many cases (and the extra buffering might smooth out any
> performance spikes due to client delays, which currently directly affect plan
> execution).
> The tricky part is managing the mismatch between the size of the row batches
> processed in {{Send()}} and the size of the fetch result asked for by the
> client. The sender materializes output rows in a {{QueryResultSet}} that is
> owned by the coordinator. That is not, currently, a splittable object -
> instead it contains the actual RPC response struct that will hit the wire
> when the RPC completes. As asynchronous sender cannot know the batch size,
> which may change on every fetch call. So the {{GetNext()}} implementation
> would need to be able to split out the {{QueryResultSet}} to match the
> correct fetch size, and handle stitching together other {{QueryResultSets}} -
> without doing extra copies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7012) NullPointerException with CTAS query
Tim Armstrong created IMPALA-7012:
-
Summary: NullPointerException with CTAS query
Key: IMPALA-7012
URL: https://issues.apache.org/jira/browse/IMPALA-7012
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Tim Armstrong
Assignee: Tianyi Wang
{noformat}
[localhost:21000] default> create table alltypesinsert partitioned by (year,
month) stored as parquet as /* +noclustered */select at1.id, at1.bool_col,
at1.tinyint_col, at1.smallint_col, at1.int_col, at1.bigint_col,
at1.float_col, at1.double_col, at1.date_string_col, at1.string_col,
at1.timestamp_col,
at1.year, at2.id as month
from functional.alltypes at1, functional.alltypes at2;
Query: create table alltypesinsert partitioned by (year, month) stored as
parquet as /* +noclustered */
select at1.id, at1.bool_col, at1.tinyint_col, at1.smallint_col, at1.int_col,
at1.bigint_col,
at1.float_col, at1.double_col, at1.date_string_col, at1.string_col,
at1.timestamp_col,
at1.year, at2.id as month
from functional.alltypes at1, functional.alltypes at2
Query submitted at: 2018-05-10 13:46:02 (Coordinator:
http://tarmstrong-box:25000)
ERROR: NullPointerException: null
{noformat}
{noformat}
I0510 13:46:02.977249 4238 Frontend.java:987] Analyzing query: create table
alltypesinsert partitioned by (year, month) stored as parquet as /*
+noclustered */
select at1.id, at1.bool_col, at1.tinyint_col, at1.smallint_col, at1.int_col,
at1.bigint_col,
at1.float_col, at1.double_col, at1.date_string_col, at1.string_col,
at1.timestamp_col,
at1.year, at2.id as month
from functional.alltypes at1, functional.alltypes at2
I0510 13:46:03.025013 4238 jni-util.cc:230] java.lang.NullPointerException
at org.apache.impala.analysis.SqlScanner.isReserved(SqlScanner.java:725)
at org.apache.impala.analysis.SqlParser.getErrorMsg(SqlParser.java:1532)
at org.apache.impala.service.Frontend.parse(Frontend.java:975)
at
org.apache.impala.service.Frontend.createExecRequest(Frontend.java:990)
at
org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:156)
I0510 13:46:03.124739 4238 status.cc:125] NullPointerException: null
@ 0x18782ef impala::Status::Status()
@ 0x1e55652 impala::JniUtil::GetJniExceptionMsg()
@ 0x1d133ed impala::JniUtil::CallJniMethod<>()
@ 0x1d10047 impala::Frontend::GetExecRequest()
@ 0x1d3205a impala::ImpalaServer::ExecuteInternal()
@ 0x1d31ba2 impala::ImpalaServer::Execute()
@ 0x1d9be70 impala::ImpalaServer::query()
@ 0x2ee378e beeswax::BeeswaxServiceProcessor::process_query()
@ 0x2ee34dc beeswax::BeeswaxServiceProcessor::dispatchCall()
@ 0x2ebcf9d impala::ImpalaServiceProcessor::dispatchCall()
@ 0x1836690 apache::thrift::TDispatchProcessor::process()
@ 0x1b9649d
apache::thrift::server::TAcceptQueueServer::Task::run()
@ 0x1b8d9c5 impala::ThriftThread::RunRunnable()
@ 0x1b8f0c9 boost::_mfi::mf2<>::operator()()
@ 0x1b8ef5f boost::_bi::list3<>::operator()<>()
@ 0x1b8ecab boost::_bi::bind_t<>::operator()()
@ 0x1b8ebbe
boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0x1bd3b1a boost::function0<>::operator()()
@ 0x1ebec51 impala::Thread::SuperviseThread()
@ 0x1ec6ded boost::_bi::list5<>::operator()<>()
@ 0x1ec6d11 boost::_bi::bind_t<>::operator()()
@ 0x1ec6cd4 boost::detail::thread_data<>::run()
@ 0x31b3a4a thread_proxy
@ 0x7fcf12d536ba start_thread
@ 0x7fcf12a8941d clone
I0510 13:46:03.124944 4238 impala-server.cc:1010] UnregisterQuery():
query_id=6b4791bb7a54de54:16bdcba7
I0510 13:46:03.124948 4238 impala-server.cc:1097] Cancel():
query_id=6b4791bb7a54de54:16bdcba7
{noformat}
This is on commit hash 3e736450354e55244e16924cfeb223a30629351d . It looks
like the code was added by IMPALA-3916
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-6970) DiskMgr::AllocateBuffersForRange crashes on failed DCHECK
[
https://issues.apache.org/jira/browse/IMPALA-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-6970.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
Impala 2.13.0
> DiskMgr::AllocateBuffersForRange crashes on failed DCHECK
> -
>
> Key: IMPALA-6970
> URL: https://issues.apache.org/jira/browse/IMPALA-6970
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.13.0
>Reporter: Sailesh Mukil
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: crash
> Fix For: Impala 2.13.0, Impala 3.1.0
>
> Attachments: stacks.txt
>
>
> Similar to IMPALA-6587, but the DCHECK failed in a slightly different way.
> Cannot tell if the root cause is the same as that though without further
> investigation.
> {code:java}
> FSF0503 09:30:26.715791 30750 reservation-tracker.cc:376] Check failed: bytes
> <= unused_reservation() (8388608 vs. 6291456)
> *** Check failure stack trace: ***
> @ 0x4277c1d google::LogMessage::Fail()
> @ 0x42794c2 google::LogMessage::SendToLog()
> @ 0x42775f7 google::LogMessage::Flush()
> @ 0x427abbe google::LogMessageFatal::~LogMessageFatal()
> @ 0x1ef1343 impala::ReservationTracker::AllocateFromLocked()
> @ 0x1ef111d impala::ReservationTracker::AllocateFrom()
> @ 0x1ee8c57
> impala::BufferPool::Client::PrepareToAllocateBuffer()
> @ 0x1ee5543 impala::BufferPool::AllocateBuffer()
> @ 0x2f50f68 impala::io::DiskIoMgr::AllocateBuffersForRange()
> @ 0x1f74762 impala::HdfsScanNodeBase::StartNextScanRange()
> @ 0x1f6b052 impala::HdfsScanNode::ScannerThread()
> @ 0x1f6a4ea
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x1f6c5cc
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x1bd4748 boost::function0<>::operator()()
> @ 0x1ebf349 impala::Thread::SuperviseThread()
> @ 0x1ec74e5 boost::_bi::list5<>::operator()<>()
> @ 0x1ec7409 boost::_bi::bind_t<>::operator()()
> @ 0x1ec73cc boost::detail::thread_data<>::run()
> @ 0x31a1f0a thread_proxy
> @ 0x36d1607851 (unknown)
> @ 0x36d12e894d (unknown)
> {code}
> Git hash of Impala used in job: ba84ad03cb83d7f7aed8524fcfbb0e2cdc9fdd53
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6979) BloomFilterBenchmark hits DCHECK
[
https://issues.apache.org/jira/browse/IMPALA-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466251#comment-16466251
]
Tim Armstrong commented on IMPALA-6979:
---
No I didn't, I assume it's reproducible though
> BloomFilterBenchmark hits DCHECK
>
>
> Key: IMPALA-6979
> URL: https://issues.apache.org/jira/browse/IMPALA-6979
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Reporter: Tim Armstrong
>Priority: Minor
>
> Leaving this here in case someone else runs into it and needs to fix the
> benchmark. We don't run this benchmark as part of builds so it's not a high
> priority to fix.
> {noformat}
> F0504 15:18:55.533821 26709 bloom-filter.cc:192] Check failed:
> !out->always_false
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-5843) Use page index in Parquet files to skip pages
[ https://issues.apache.org/jira/browse/IMPALA-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong reassigned IMPALA-5843: - Assignee: Zoltán Borók-Nagy > Use page index in Parquet files to skip pages > - > > Key: IMPALA-5843 > URL: https://issues.apache.org/jira/browse/IMPALA-5843 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Affects Versions: Impala 2.10.0 >Reporter: Lars Volker >Assignee: Zoltán Borók-Nagy >Priority: Critical > Labels: parquet, performance > > Once IMPALA-5842 has been resolved, we should skip pages based on the page > index in Parquet files. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7030) crash in impala::PartitionedAggregationNode::ProcessBatchNoGrouping
[
https://issues.apache.org/jira/browse/IMPALA-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16476076#comment-16476076
]
Tim Armstrong commented on IMPALA-7030:
---
I also saw a crash in codegen'd code on the same job. I wasn't able to get a
backtrace but I reconfigured the job to collect the hs_err_pid* file in future.
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2183/
{noformat}
Crash reason: SIGSEGV
Crash address: 0x7f87bd39d000
Process uptime: not available
Thread 223 (crashed)
0 0x7f87bd39d000
rax = 0x7f87bd39d000 rdx = 0x0ec12000
rcx = 0x0f547540 rbx = 0x
rsi = 0x0f547540 rdi = 0x0ec12000
rbp = 0x7f8767575850 rsp = 0x7f8767575658
r8 = 0x0f7255a0r9 = 0x7f87675751ef
r10 = 0x17966000 r11 = 0x0020
r12 = 0x0400 r13 = 0x0011
r14 = 0x2d2a9000 r15 = 0x0f549210
rip = 0x7f87bd39d000
Found by: given as instruction pointer in context
1 impalad + 0x1961f63
rbp = 0x7f8767575b20 rsp = 0x7f8767575860
rip = 0x01d61f63
Found by: previous frame's frame pointer
2 impalad + 0x195e6cf
rbp = 0x7f87675760e0 rsp = 0x7f8767575b30
rip = 0x01d5e6cf
Found by: previous frame's frame pointer
3 impalad + 0x195c89e
rbp = 0x7f87675761a0 rsp = 0x7f87675760f0
rip = 0x01d5c89e
Found by: previous frame's frame pointer
4 impalad + 0x18e42c8
rbp = 0x7f8767576510 rsp = 0x7f87675761b0
rip = 0x01ce42c8
Found by: previous frame's frame pointer
5 impalad + 0x18e3669
rbp = 0x7f87675768c0 rsp = 0x7f8767576520
rip = 0x01ce3669
Found by: previous frame's frame pointer
6 impalad + 0x18e2adc
rbp = 0x7f87675768e0 rsp = 0x7f87675768d0
rip = 0x01ce2adc
Found by: previous frame's frame pointer
Crash reason: SIGSEGV
Crash address: 0x7f87bd39d000
Process uptime: not available
Thread 223 (crashed)
0 0x7f87bd39d000
rax = 0x7f87bd39d000 rdx = 0x0ec12000
rcx = 0x0f547540 rbx = 0x
rsi = 0x0f547540 rdi = 0x0ec12000
rbp = 0x7f8767575850 rsp = 0x7f8767575658
r8 = 0x0f7255a0r9 = 0x7f87675751ef
r10 = 0x17966000 r11 = 0x0020
r12 = 0x0400 r13 = 0x0011
r14 = 0x2d2a9000 r15 = 0x0f549210
rip = 0x7f87bd39d000
Found by: given as instruction pointer in context
1 impalad + 0x1961f63
rbp = 0x7f8767575b20 rsp = 0x7f8767575860
rip = 0x01d61f63
Found by: previous frame's frame pointer
2 impalad + 0x195e6cf
rbp = 0x7f87675760e0 rsp = 0x7f8767575b30
rip = 0x01d5e6cf
Found by: previous frame's frame pointer
3 impalad + 0x195c89e
rbp = 0x7f87675761a0 rsp = 0x7f87675760f0
rip = 0x01d5c89e
Found by: previous frame's frame pointer
4 impalad + 0x18e42c8
rbp = 0x7f8767576510 rsp = 0x7f87675761b0
rip = 0x01ce42c8
Found by: previous frame's frame pointer
5 impalad + 0x18e3669
rbp = 0x7f87675768c0 rsp = 0x7f8767576520
rip = 0x01ce3669
Found by: previous frame's frame pointer
6 impalad + 0x18e2adc
rbp = 0x7f87675768e0 rsp = 0x7f87675768d0
rip = 0x01ce2adc
Found by: previous frame's frame pointer
Crash reason: SIGSEGV
Crash address: 0x7f87bd39d000
Process uptime: not available
Thread 223 (crashed)
0 0x7f87bd39d000
rax = 0x7f87bd39d000 rdx = 0x0ec12000
rcx = 0x0f547540 rbx = 0x
rsi = 0x0f547540 rdi = 0x0ec12000
rbp = 0x7f8767575850 rsp = 0x7f8767575658
r8 = 0x0f7255a0r9 = 0x7f87675751ef
r10 = 0x17966000 r11 = 0x0020
r12 = 0x0400 r13 = 0x0011
r14 = 0x2d2a9000 r15 = 0x0f549210
rip = 0x7f87bd39d000
Found by: given as instruction pointer in context
1 impalad + 0x1961f63
rbp = 0x7f8767575b20 rsp = 0x7f8767575860
rip = 0x01d61f63
Found by: previous frame's frame pointer
2 impalad + 0x195e6cf
rbp = 0x7f87675760e0 rsp = 0x7f8767575b30
rip = 0x01d5e6cf
Found by: previous frame's frame pointer
3 impalad + 0x195c89e
rbp = 0x7f87675761a0 rsp = 0x7f87675760f0
rip = 0x01d5c89e
Found by: previous frame's frame pointer
4 impalad + 0x18e42c8
rbp = 0x7f8767576510 rsp = 0x7f87675761b0
rip = 0x01ce42c8
Found by: previous frame's frame pointer
5 impalad + 0x18e3669
rbp = 0x7f87675768c0 rsp = 0x7f8767576520
rip =
[jira] [Commented] (IMPALA-7031) Detailed failure reason for clients when a query is cancelled from the WebUi
[ https://issues.apache.org/jira/browse/IMPALA-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16476046#comment-16476046 ] Tim Armstrong commented on IMPALA-7031: --- [~adrenas] I think this is only possible if the query is "closed" from the web UI, not just cancelled. If you can reproduce with the query still in the "waiting to be closed" state then something weird is going on. > Detailed failure reason for clients when a query is cancelled from the WebUi > > > Key: IMPALA-7031 > URL: https://issues.apache.org/jira/browse/IMPALA-7031 > Project: IMPALA > Issue Type: New Feature >Reporter: Adriano >Priority: Major > > In big clusters with many jdbc/odbc users, in order to save resources are > often implemented scripts that automatically cancel queries (e.g. long > running queries) (the scripts typically are using the Impala Webui). > Typical Scenario: > # A jdbc/odbc client submit a query > # The Coordinator start the query execution > # The query is cancelled from the Coordinator WebUi > # The jdbc/odbc client ask to the Coordinator the query status > (GetOperationStatus) > # The Coordinator answer "unknown query ID" (as the query was cancelled) > # For the client perspective the query failed for "unknown query ID" > Currently, if a running query is cancelled from the impalad WebUI, the client > will just receive an 'unknown query ID' error on the next > fetch/getOperationStatus attempt. It would be good to be able to explicitly > call out this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7032) Codegen crash when UNIONing NULL and CHAR(N)
[
https://issues.apache.org/jira/browse/IMPALA-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7032:
--
Affects Version/s: Impala 3.1.0
> Codegen crash when UNIONing NULL and CHAR(N)
>
>
> Key: IMPALA-7032
> URL: https://issues.apache.org/jira/browse/IMPALA-7032
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.9.0, Impala 2.10.0, Impala 2.11.0, Impala 3.0,
> Impala 2.12.0, Impala 2.13.0, Impala 3.1.0
>Reporter: Balazs Jeszenszky
>Priority: Blocker
> Labels: crash
>
> A simple repro:
> {code:java}
> create table test (c1 int);
> select null from test union select cast('a' as char(1)) from test;
> {code}
> {code}
> #0 0x7f050c4a61d7 in raise () from sysroot/lib64/libc.so.6
> #1 0x7f050c4a78c8 in abort () from sysroot/lib64/libc.so.6
> #2 0x7f050e7816b5 in os::abort(bool) ()
> from sysroot/usr/java/jdk1.8.0_60/jre/lib/amd64/server/libjvm.so
> #3 0x7f050e91fbf3 in VMError::report_and_die() ()
> from sysroot/usr/java/jdk1.8.0_60/jre/lib/amd64/server/libjvm.so
> #4 0x7f050e786edf in JVM_handle_linux_signal ()
> from sysroot/usr/java/jdk1.8.0_60/jre/lib/amd64/server/libjvm.so
> #5 0x7f050e77d673 in signalHandler(int, siginfo*, void*) ()
> from sysroot/usr/java/jdk1.8.0_60/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x01a9123d in llvm::FunctionType::get(llvm::Type*,
> llvm::ArrayRef, bool) ()
> #8 0x00c2c04f in impala::LlvmCodeGen::FnPrototype::GeneratePrototype
> (this=0x7f03e7fdad90,
> builder=0x0, params=0x7f03e7fdae30, print_ir=)
> at
> /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/codegen/llvm-codegen.cc:710
> #9 0x00846187 in impala::Expr::CreateIrFunctionPrototype
> (this=this@entry=0x9fded80,
> codegen=codegen@entry=0xa5a2880, name=..., args=args@entry=0x7f03e7fdae30)
> at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/expr.cc:505
> #10 0x00861e5c in impala::NullLiteral::GetCodegendComputeFn
> (this=0x9fded80,
> codegen=0xa5a2880, fn=0x7f03e7fdaf18)
> at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/null-literal.cc:106
> #11 0x00a79bc7 in impala::Tuple::CodegenMaterializeExprs
> (codegen=codegen@entry=0xa5a2880,
> collect_string_vals=collect_string_vals@entry=false, desc=...,
> materialize_expr_ctxs=...,
> use_mem_pool=use_mem_pool@entry=true, fn=0x7f03e7fdb410)
> at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/runtime/tuple.cc:307
> #12 0x00d12828 in impala::UnionNode::Codegen (this=,
> state=)
> at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exec/union-node.cc:105
> #13 0x00c4aaa1 in impala::ExecNode::Codegen
> (this=this@entry=0xa0c5480,
> {code}
> NullLiteral::GetCodegendComputeFn is missing a check for type.CHAR which
> isn't implemented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7000) Wrong info about Impala dedicated executors
[ https://issues.apache.org/jira/browse/IMPALA-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-7000: -- Summary: Wrong info about Impala dedicated executors (was: Wrong info about Impala dedicated executore) > Wrong info about Impala dedicated executors > --- > > Key: IMPALA-7000 > URL: https://issues.apache.org/jira/browse/IMPALA-7000 > Project: IMPALA > Issue Type: Bug > Components: Docs >Affects Versions: Impala 2.12.0 >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > > The following is not correct. > "Then, you specify that the other hosts act as executors but not > coordinators. These hosts do not communicate with the statestored daemon or > process the final result sets from queries. You cannot connect to these hosts > through clients such as impala-shell or business intelligence tools." > executor still communicates with statestore for other topics (membership, > admission control, etc.) The only part it doesn't get from statestore is the > metadata topic. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-6645) Consider enabling disk spill encryption by default
[ https://issues.apache.org/jira/browse/IMPALA-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-6645. --- Resolution: Fixed Fix Version/s: Impala 3.1.0 Impala 2.13.0 > Consider enabling disk spill encryption by default > -- > > Key: IMPALA-6645 > URL: https://issues.apache.org/jira/browse/IMPALA-6645 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Not Applicable >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: resource-management > Fix For: Impala 2.13.0, Impala 3.1.0 > > > We currently have integrity checks disabled by default because of the > additional CPU cost. > Now that IMPALA-6219 is in, the integrity checks are cheaper so it may be > worth just paying the cost. We could also consider doing a lighter-weight CRC > checksum, but AES-GCM may be fast enough at 2.3GB/s to do by default. > We should do an experiment to see if there's a major impact on spill-to-disk > perf. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-6035) Add query option that rejects queries based on query complexity
[ https://issues.apache.org/jira/browse/IMPALA-6035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-6035 started by Tim Armstrong. - > Add query option that rejects queries based on query complexity > --- > > Key: IMPALA-6035 > URL: https://issues.apache.org/jira/browse/IMPALA-6035 > Project: IMPALA > Issue Type: Sub-task > Components: Distributed Exec >Reporter: Mostafa Mokhtar >Assignee: Tim Armstrong >Priority: Major > Labels: admission-control > > Expose query option to reject queries with a large number of remote > fragments. > Queries with a large number of fragments can negatively affect cluster > behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7044) int32 overflow in HdfsTableSink::CreateNewTmpFile()
[
https://issues.apache.org/jira/browse/IMPALA-7044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482642#comment-16482642
]
Tim Armstrong commented on IMPALA-7044:
---
[~lv] maybe we should just cap the number of columns that we support writing? I
don't think it makes sense to support an unlimited number of columns for
inserts - at some point we should start putting backpressure on users before
they push the system outside of the limits where it will behave well.
> int32 overflow in HdfsTableSink::CreateNewTmpFile()
> ---
>
> Key: IMPALA-7044
> URL: https://issues.apache.org/jira/browse/IMPALA-7044
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.9.0, Impala 2.10.0, Impala 2.11.0, Impala 3.0,
> Impala 2.12.0, Impala 2.13.0
>Reporter: Lars Volker
>Priority: Critical
> Labels: parquet
> Attachments: ct.sql
>
>
> When writing Parquet files we compute a minimum block size based on the
> number of columns in the target table in
> [hdfs-parquet-table-writer.cc:916|https://github.com/apache/impala/blob/master/be/src/exec/hdfs-parquet-table-writer.cc?utf8=%E2%9C%93#L916]:
> {noformat}
> 3 * DEFAULT_DATA_PAGE_SIZE * columns_.size();
> {noformat}
> For tables with a large number of columns (> ~10k), this value will get
> larger than 2GB. When we pass it to {{hdfsOpenFile()}} in
> {{HdfsTableSink::CreateNewTmpFile()}} it gets cast to a signed int32 and can
> overflow.
> This leads to error messages like the following:
> {noformat}
> I0516 16:13:52.755090 24257 status.cc:125] Failed to open HDFS file for
> writing:
> hdfs://localhost:20500/test-warehouse/lv.db/a/_impala_insert_staging/3c417cb973b710ab_803e8980/.3c417cb973b710ab-80
> 3e8980_411033576_dir/3c417cb973b710ab-803e8980_271567064_data.0.parq
> Error(255): Unknown error 255
> Root cause: RemoteException: Specified block size is less than configured
> minimum value (dfs.namenode.fs-limits.min-block-size): -1935671296 < 1024
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2417)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2339)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:764)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> @ 0x187b8b3 impala::Status::Status()
> @ 0x1fade89 impala::HdfsTableSink::CreateNewTmpFile()
> @ 0x1faeee7 impala::HdfsTableSink::InitOutputPartition()
> @ 0x1fb1389 impala::HdfsTableSink::GetOutputPartition()
> @ 0x1faf34a impala::HdfsTableSink::Send()
> @ 0x1c91bcd impala::FragmentInstanceState::ExecInternal()
> @ 0x1c8efa5 impala::FragmentInstanceState::Exec()
> @ 0x1c9e53f impala::QueryState::ExecFInstance()
> @ 0x1c9cdb2
> _ZZN6impala10QueryState15StartFInstancesEvENKUlvE_clEv
> @ 0x1c9f25d
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala10QueryState15StartFInstancesEvEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x1bd6cd4 boost::function0<>::operator()()
> @ 0x1ec18f9 impala::Thread::SuperviseThread()
> @ 0x1ec9a95 boost::_bi::list5<>::operator()<>()
> @ 0x1ec99b9 boost::_bi::bind_t<>::operator()()
> @ 0x1ec997c boost::detail::thread_data<>::run()
> @ 0x31a527a thread_proxy
> @ 0x7f30246a8184 start_thread
> @ 0x7f30243d503d clone
> {noformat}
> The signature of {{hdfsOpenFile()}} is as follows:
> {noformat}
> hdfsFile hdfsOpenFile(hdfsFS fs, const char* path, int flags, int bufferSize,
> short replication, tSize blocksize);
> {noformat}
> {{tSize}} is typedef'd to {{int32_t}}.
> The comment of {{hdfsOpenFile()}} is explicit about this:
>
[jira] [Created] (IMPALA-7053) Reorganise query options into groups
Tim Armstrong created IMPALA-7053: - Summary: Reorganise query options into groups Key: IMPALA-7053 URL: https://issues.apache.org/jira/browse/IMPALA-7053 Project: IMPALA Issue Type: Improvement Components: Backend Reporter: Tim Armstrong Assignee: Tim Armstrong We have quite a lot of query options now and we're adding more for things like resource limits (e.g. IMPALA-6035). It's getting harder for users to understand the organisation and find relevant query options. We should consider grouping similar query options. E.g. for this set of resource limits, we could reorganise in various ways: * mem_limit -> resources.memory.per_node_limit * buffer_pool_limit -> resources.memory.buffer_pool.per_node_limit * thread_reservation_limit -> resources.threads.per_node_limit * thread_reservation_aggregate_limit -> resources.threads.aggregate_limit * exec_time_limit_s -> resources.wallclock.limit_s We could do the conversion incrementally. It would probably make sense to agree on a top-level organisation up-front. * planner - anything that controls planner decisions like join ordering, etc * scheduler - anything that controls scheduler decisions (admission control could maybe be included here too) * resources - resource management functionality (limits, etc) * session - anything related to session management like timeouts * exec - anything that changes query execution behaviour (e.g. codegen, batch sizes, runtime filters, etc) * Probably a group for anything that changes the semantic behaviour of a query (e.g. decimal_v2, appx_count_distinct, strict_mode, abort_on_error). * A group that controls read and write behaviour of file formats like compression, etc -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7044) int32 overflow in HdfsTableSink::CreateNewTmpFile()
[
https://issues.apache.org/jira/browse/IMPALA-7044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482775#comment-16482775
]
Tim Armstrong commented on IMPALA-7044:
---
Yeah, so regardless I think we should pick an upper limit and make sure that we
test up to that limit to be sure it works well as part of fixing this bug.
> int32 overflow in HdfsTableSink::CreateNewTmpFile()
> ---
>
> Key: IMPALA-7044
> URL: https://issues.apache.org/jira/browse/IMPALA-7044
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.9.0, Impala 2.10.0, Impala 2.11.0, Impala 3.0,
> Impala 2.12.0, Impala 2.13.0
>Reporter: Lars Volker
>Priority: Critical
> Labels: parquet
> Attachments: ct.sql
>
>
> When writing Parquet files we compute a minimum block size based on the
> number of columns in the target table in
> [hdfs-parquet-table-writer.cc:916|https://github.com/apache/impala/blob/master/be/src/exec/hdfs-parquet-table-writer.cc?utf8=%E2%9C%93#L916]:
> {noformat}
> 3 * DEFAULT_DATA_PAGE_SIZE * columns_.size();
> {noformat}
> For tables with a large number of columns (> ~10k), this value will get
> larger than 2GB. When we pass it to {{hdfsOpenFile()}} in
> {{HdfsTableSink::CreateNewTmpFile()}} it gets cast to a signed int32 and can
> overflow.
> This leads to error messages like the following:
> {noformat}
> I0516 16:13:52.755090 24257 status.cc:125] Failed to open HDFS file for
> writing:
> hdfs://localhost:20500/test-warehouse/lv.db/a/_impala_insert_staging/3c417cb973b710ab_803e8980/.3c417cb973b710ab-80
> 3e8980_411033576_dir/3c417cb973b710ab-803e8980_271567064_data.0.parq
> Error(255): Unknown error 255
> Root cause: RemoteException: Specified block size is less than configured
> minimum value (dfs.namenode.fs-limits.min-block-size): -1935671296 < 1024
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2417)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2339)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:764)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> @ 0x187b8b3 impala::Status::Status()
> @ 0x1fade89 impala::HdfsTableSink::CreateNewTmpFile()
> @ 0x1faeee7 impala::HdfsTableSink::InitOutputPartition()
> @ 0x1fb1389 impala::HdfsTableSink::GetOutputPartition()
> @ 0x1faf34a impala::HdfsTableSink::Send()
> @ 0x1c91bcd impala::FragmentInstanceState::ExecInternal()
> @ 0x1c8efa5 impala::FragmentInstanceState::Exec()
> @ 0x1c9e53f impala::QueryState::ExecFInstance()
> @ 0x1c9cdb2
> _ZZN6impala10QueryState15StartFInstancesEvENKUlvE_clEv
> @ 0x1c9f25d
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala10QueryState15StartFInstancesEvEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x1bd6cd4 boost::function0<>::operator()()
> @ 0x1ec18f9 impala::Thread::SuperviseThread()
> @ 0x1ec9a95 boost::_bi::list5<>::operator()<>()
> @ 0x1ec99b9 boost::_bi::bind_t<>::operator()()
> @ 0x1ec997c boost::detail::thread_data<>::run()
> @ 0x31a527a thread_proxy
> @ 0x7f30246a8184 start_thread
> @ 0x7f30243d503d clone
> {noformat}
> The signature of {{hdfsOpenFile()}} is as follows:
> {noformat}
> hdfsFile hdfsOpenFile(hdfsFS fs, const char* path, int flags, int bufferSize,
> short replication, tSize blocksize);
> {noformat}
> {{tSize}} is typedef'd to {{int32_t}}.
> The comment of {{hdfsOpenFile()}} is explicit about this:
> {noformat}
> @param blocksize Size of block - pass 0 if you want to use the
> default configured values. Note that if you want a block size bigger
[jira] [Resolved] (IMPALA-7057) crash when DiskIoMgr::GetFreeBuffer(long*)+0xa9
[ https://issues.apache.org/jira/browse/IMPALA-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-7057. --- Resolution: Duplicate We changed the code to prevent the crash in Impala 2.7.0. I believe that the crash in that place most frequently required a corrupt file to trigger it (a negative page length would do it) - we tightened up the handling of that over several releases, notably with IMPALA-3917 in Impala 2.7.0 for Parquet > crash when DiskIoMgr::GetFreeBuffer(long*)+0xa9 > --- > > Key: IMPALA-7057 > URL: https://issues.apache.org/jira/browse/IMPALA-7057 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 2.5.0 >Reporter: KarlManong >Priority: Major > Attachments: hs_err_pid1495848.log > > > impalad version 2.5.0-cdh5.7.1 RELEASE (build > 27a4325c18c2a01c7a8097681a0eccf6d4335ea1)。 > some query will cause impalad crashed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-3544) crash in DiskIoMgr::GetFreeBuffer
[
https://issues.apache.org/jira/browse/IMPALA-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-3544:
--
Summary: crash in DiskIoMgr::GetFreeBuffer (was: data-stream-sender.cc:261
channel send status: Sender timed out waiting for receiver fragment instance:
a045d3abcd1bed4f:a116563837c8ebda)
> crash in DiskIoMgr::GetFreeBuffer
> -
>
> Key: IMPALA-3544
> URL: https://issues.apache.org/jira/browse/IMPALA-3544
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.5.0
> Environment: Ubuntu Trusty, CDH 5.7.0
>Reporter: Scott Wallace
>Assignee: Sailesh Mukil
>Priority: Critical
> Labels: crash, hang
> Attachments: 2016-05-17_1441.png, 2016-05-18_1109.png,
> 2016-05-18_1140.png, 2016-05-26_1438.png, 2016-05-26_1440.png, Screen Shot
> 2016-05-27 at 12.43.23 PM.png, hs_err_pid7016.log,
> impalad.ux-reporting-engine-worker-13-prod-us-east-1a.impala.log.ERROR.20160516-152100.89097,
>
> impalad.ux-reporting-engine-worker-18-prod-us-east-1a.impala.log.ERROR.20160519-140954.7016,
>
> impalad.ux-reporting-engine-worker-6-prod-us-east-1a.impala.log.INFO.20160516-152100.44312
>
>
> Impala becomes unresponsive, causing a full outage for our reporting system.
> Impalad logs show various errors as follows:
> {noformat}
> Time Log Level Source Log Message
> May 16, 3:20:59.953 PMERROR logging.cc:120
> stderr will be logged to this file.
> May 16, 3:27:38.652 PMERROR data-stream-sender.cc:261
> channel send status:
> Sender timed out waiting for receiver fragment instance:
> a045d3abcd1bed4f:a116563837c8ebda
> May 16, 3:27:39.738 PMERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:44.784 PMERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:46.055 PMERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:46.371 PMERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:46.906 PMERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:51.003 PMERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:52.479 PMERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:54.227 PMERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:54.298 PMERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:54.305 PMERROR data-stream-sender.cc:261
> channel send status:
> Sender timed out waiting for receiver fragment instance:
> 7446d34d9eefb9f:436f61bcd3c78eb3
> May 16, 3:27:55.022 PMERROR data-stream-sender.cc:261
> channel send status:
> Sender timed out waiting for receiver fragment instance:
> b5bcb782b5a4:5cef13b26af20aa1
> May 16, 3:27:55.646 PMERROR data-stream-sender.cc:261
> channel send status:
> Sender timed out waiting for receiver fragment instance:
> 3249c6494703ecc7:3b9186fbed00dc
> May 16, 3:27:56.042 PMERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:56.277 PMERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport for
> ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed:
> Connection timed out)
> May 16, 3:27:56.749 PMERROR data-stream-sender.cc:261
> channel send status: Couldn't open transport
[jira] [Commented] (IMPALA-7055) test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to table format AVRO is not supported")
[
https://issues.apache.org/jira/browse/IMPALA-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488020#comment-16488020
]
Tim Armstrong commented on IMPALA-7055:
---
I can repro by adding a sleep:
{code:java}
if (backend_state->ApplyExecStatusReport(params, _summary_, _))
{
// This backend execution has completed.
bool is_fragment_failure;
TUniqueId failed_instance_id;
Status status = backend_state->GetStatus(_fragment_failure,
_instance_id);
int pending_backends = backend_exec_complete_barrier_->Notify();
if (VLOG_QUERY_IS_ON && pending_backends >= 0) {
VLOG_QUERY << "Backend completed:"
<< " host=" <<
TNetworkAddressToString(backend_state->impalad_address())
<< " remaining=" << pending_backends
<< " query_id=" << PrintId(query_id());
BackendState::LogFirstInProgress(backend_states_);
}
SleepForMs(1000);
if (!status.ok()) {
// We may start receiving status reports before all exec rpcs are
complete.
// Can't apply state transition until no more exec rpcs will be sent.
exec_rpcs_complete_barrier_->Wait();
discard_result(UpdateExecState(status,
is_fragment_failure ? _instance_id : nullptr,
TNetworkAddressToString(backend_state->impalad_address(;
}
}{code}
> test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to
> table format AVRO is not supported")
> --
>
> Key: IMPALA-7055
> URL: https://issues.apache.org/jira/browse/IMPALA-7055
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0
>Reporter: David Knupp
>Assignee: Tim Armstrong
>Priority: Major
> Labels: flaky
>
> This failure occurred while verifying https://gerrit.cloudera.org/c/10455/,
> but it is not related to that patch. The failing build is
> https://jenkins.impala.io/job/gerrit-verify-dryrun/2511/
> (https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2232/)
> Test appears to be (from
> [avro-writer.test|https://github.com/apache/impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/avro-writer.test]):
> {noformat}
> QUERY
> SET ALLOW_UNSUPPORTED_FORMATS=0;
> insert into __avro_write select 1, "b", 2.2;
> CATCH
> Writing to table format AVRO is not supported. Use query option
> ALLOW_UNSUPPORTED_FORMATS
> {noformat}
> Error output:
> {noformat}
> 01:50:18 ] FAIL
> query_test/test_compressed_formats.py::TestTableWriters::()::test_avro_writer[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> 01:50:18 ] === FAILURES
> ===
> 01:50:18 ] TestTableWriters.test_avro_writer[exec_option: {'batch_size': 0,
> 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen':
> False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> 01:50:18 ] [gw9] linux2 -- Python 2.7.12
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 01:50:18 ] query_test/test_compressed_formats.py:189: in test_avro_writer
> 01:50:18 ] self.run_test_case('QueryTest/avro-writer', vector)
> 01:50:18 ] common/impala_test_suite.py:420: in run_test_case
> 01:50:18 ] assert False, "Expected exception: %s" % expected_str
> 01:50:18 ] E AssertionError: Expected exception: Writing to table format
> AVRO is not supported. Use query option ALLOW_UNSUPPORTED_FORMATS
> 01:50:18 ] Captured stderr setup
> -
> 01:50:18 ] -- connecting to: localhost:21000
> 01:50:18 ] - Captured stderr call
> -
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] use functional;
> 01:50:18 ]
> 01:50:18 ] SET batch_size=0;
> 01:50:18 ] SET num_nodes=0;
> 01:50:18 ] SET disable_codegen_rows_threshold=5000;
> 01:50:18 ] SET disable_codegen=False;
> 01:50:18 ] SET abort_on_error=1;
> 01:50:18 ] SET exec_single_node_rows_threshold=0;
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] drop table if exists __avro_write;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=NONE;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] create table __avro_write (i int, s string, d double)
> 01:50:18 ] stored as AVRO
> 01:50:18 ] TBLPROPERTIES ('avro.schema.literal'='{
> 01:50:18 ] "name": "my_record",
>
[jira] [Updated] (IMPALA-7055) test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to table format AVRO is not supported")
[
https://issues.apache.org/jira/browse/IMPALA-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7055:
--
Target Version: Impala 3.1.0
> test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to
> table format AVRO is not supported")
> --
>
> Key: IMPALA-7055
> URL: https://issues.apache.org/jira/browse/IMPALA-7055
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0
>Reporter: David Knupp
>Assignee: Tim Armstrong
>Priority: Major
> Labels: flaky
>
> This failure occurred while verifying https://gerrit.cloudera.org/c/10455/,
> but it is not related to that patch. The failing build is
> https://jenkins.impala.io/job/gerrit-verify-dryrun/2511/
> (https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2232/)
> Test appears to be (from
> [avro-writer.test|https://github.com/apache/impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/avro-writer.test]):
> {noformat}
> QUERY
> SET ALLOW_UNSUPPORTED_FORMATS=0;
> insert into __avro_write select 1, "b", 2.2;
> CATCH
> Writing to table format AVRO is not supported. Use query option
> ALLOW_UNSUPPORTED_FORMATS
> {noformat}
> Error output:
> {noformat}
> 01:50:18 ] FAIL
> query_test/test_compressed_formats.py::TestTableWriters::()::test_avro_writer[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> 01:50:18 ] === FAILURES
> ===
> 01:50:18 ] TestTableWriters.test_avro_writer[exec_option: {'batch_size': 0,
> 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen':
> False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> 01:50:18 ] [gw9] linux2 -- Python 2.7.12
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 01:50:18 ] query_test/test_compressed_formats.py:189: in test_avro_writer
> 01:50:18 ] self.run_test_case('QueryTest/avro-writer', vector)
> 01:50:18 ] common/impala_test_suite.py:420: in run_test_case
> 01:50:18 ] assert False, "Expected exception: %s" % expected_str
> 01:50:18 ] E AssertionError: Expected exception: Writing to table format
> AVRO is not supported. Use query option ALLOW_UNSUPPORTED_FORMATS
> 01:50:18 ] Captured stderr setup
> -
> 01:50:18 ] -- connecting to: localhost:21000
> 01:50:18 ] - Captured stderr call
> -
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] use functional;
> 01:50:18 ]
> 01:50:18 ] SET batch_size=0;
> 01:50:18 ] SET num_nodes=0;
> 01:50:18 ] SET disable_codegen_rows_threshold=5000;
> 01:50:18 ] SET disable_codegen=False;
> 01:50:18 ] SET abort_on_error=1;
> 01:50:18 ] SET exec_single_node_rows_threshold=0;
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] drop table if exists __avro_write;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=NONE;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] create table __avro_write (i int, s string, d double)
> 01:50:18 ] stored as AVRO
> 01:50:18 ] TBLPROPERTIES ('avro.schema.literal'='{
> 01:50:18 ] "name": "my_record",
> 01:50:18 ] "type": "record",
> 01:50:18 ] "fields": [
> 01:50:18 ] {"name":"i", "type":["int", "null"]},
> 01:50:18 ] {"name":"s", "type":["string", "null"]},
> 01:50:18 ] {"name":"d", "type":["double", "null"]}]}');
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC="";
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=NONE;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS=1;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] insert into __avro_write select 0, "a", 1.1;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC="";
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS="0";
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=SNAPPY;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
>
[jira] [Updated] (IMPALA-7055) test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to table format AVRO is not supported")
[
https://issues.apache.org/jira/browse/IMPALA-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7055:
--
Priority: Blocker (was: Major)
> test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to
> table format AVRO is not supported")
> --
>
> Key: IMPALA-7055
> URL: https://issues.apache.org/jira/browse/IMPALA-7055
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0
>Reporter: David Knupp
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: flaky
>
> This failure occurred while verifying https://gerrit.cloudera.org/c/10455/,
> but it is not related to that patch. The failing build is
> https://jenkins.impala.io/job/gerrit-verify-dryrun/2511/
> (https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2232/)
> Test appears to be (from
> [avro-writer.test|https://github.com/apache/impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/avro-writer.test]):
> {noformat}
> QUERY
> SET ALLOW_UNSUPPORTED_FORMATS=0;
> insert into __avro_write select 1, "b", 2.2;
> CATCH
> Writing to table format AVRO is not supported. Use query option
> ALLOW_UNSUPPORTED_FORMATS
> {noformat}
> Error output:
> {noformat}
> 01:50:18 ] FAIL
> query_test/test_compressed_formats.py::TestTableWriters::()::test_avro_writer[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> 01:50:18 ] === FAILURES
> ===
> 01:50:18 ] TestTableWriters.test_avro_writer[exec_option: {'batch_size': 0,
> 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen':
> False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> 01:50:18 ] [gw9] linux2 -- Python 2.7.12
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 01:50:18 ] query_test/test_compressed_formats.py:189: in test_avro_writer
> 01:50:18 ] self.run_test_case('QueryTest/avro-writer', vector)
> 01:50:18 ] common/impala_test_suite.py:420: in run_test_case
> 01:50:18 ] assert False, "Expected exception: %s" % expected_str
> 01:50:18 ] E AssertionError: Expected exception: Writing to table format
> AVRO is not supported. Use query option ALLOW_UNSUPPORTED_FORMATS
> 01:50:18 ] Captured stderr setup
> -
> 01:50:18 ] -- connecting to: localhost:21000
> 01:50:18 ] - Captured stderr call
> -
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] use functional;
> 01:50:18 ]
> 01:50:18 ] SET batch_size=0;
> 01:50:18 ] SET num_nodes=0;
> 01:50:18 ] SET disable_codegen_rows_threshold=5000;
> 01:50:18 ] SET disable_codegen=False;
> 01:50:18 ] SET abort_on_error=1;
> 01:50:18 ] SET exec_single_node_rows_threshold=0;
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] drop table if exists __avro_write;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=NONE;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] create table __avro_write (i int, s string, d double)
> 01:50:18 ] stored as AVRO
> 01:50:18 ] TBLPROPERTIES ('avro.schema.literal'='{
> 01:50:18 ] "name": "my_record",
> 01:50:18 ] "type": "record",
> 01:50:18 ] "fields": [
> 01:50:18 ] {"name":"i", "type":["int", "null"]},
> 01:50:18 ] {"name":"s", "type":["string", "null"]},
> 01:50:18 ] {"name":"d", "type":["double", "null"]}]}');
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC="";
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=NONE;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS=1;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] insert into __avro_write select 0, "a", 1.1;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC="";
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS="0";
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=SNAPPY;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
>
[jira] [Assigned] (IMPALA-6998) test_bloom_wait_time fails due to late arrival of filters on Isilon
[
https://issues.apache.org/jira/browse/IMPALA-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-6998:
-
Assignee: Tim Armstrong
> test_bloom_wait_time fails due to late arrival of filters on Isilon
> ---
>
> Key: IMPALA-6998
> URL: https://issues.apache.org/jira/browse/IMPALA-6998
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Sailesh Mukil
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build
>
> This is likely a flaky issue and was seen on an instance of an Isilon run:
> {code:java}
> Error Message
> query_test/test_runtime_filters.py:92: in test_bloom_wait_time assert
> duration < 60, \ E AssertionError: Query took too long (118.044356108s,
> possibly waiting for missing filters?) E assert 118.04435610771179 < 60
> Stacktrace
> query_test/test_runtime_filters.py:92: in test_bloom_wait_time
> assert duration < 60, \
> E AssertionError: Query took too long (118.044356108s, possibly waiting for
> missing filters?)
> E assert 118.04435610771179 < 60
> Standard Error
> -- executing against localhost:21000
> use functional_parquet;
> SET batch_size=0;
> SET num_nodes=0;
> SET disable_codegen_rows_threshold=0;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> -- executing against localhost:21000
> SET RUNTIME_FILTER_WAIT_TIME_MS=60;
> -- executing against localhost:21000
> SET RUNTIME_FILTER_MODE=GLOBAL;
> -- executing against localhost:21000
> SET RUNTIME_FILTER_MAX_SIZE=64K;
> -- executing against localhost:21000
> with l as (select * from tpch.lineitem UNION ALL select * from tpch.lineitem)
> select STRAIGHT_JOIN count(*) from (select * from tpch.lineitem a LIMIT 1) a
> join (select * from l LIMIT 50) b on a.l_orderkey = -b.l_orderkey;
> -- executing against localhost:21000
> SET RUNTIME_FILTER_WAIT_TIME_MS="0";
> -- executing against localhost:21000
> SET RUNTIME_FILTER_MODE="GLOBAL";
> -- executing against localhost:21000
> SET RUNTIME_FILTER_MAX_SIZE="16777216";
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6998) test_bloom_wait_time fails due to late arrival of filters on Isilon
[
https://issues.apache.org/jira/browse/IMPALA-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-6998:
-
Assignee: Sailesh Mukil (was: Tim Armstrong)
> test_bloom_wait_time fails due to late arrival of filters on Isilon
> ---
>
> Key: IMPALA-6998
> URL: https://issues.apache.org/jira/browse/IMPALA-6998
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Sailesh Mukil
>Assignee: Sailesh Mukil
>Priority: Critical
> Labels: broken-build
>
> This is likely a flaky issue and was seen on an instance of an Isilon run:
> {code:java}
> Error Message
> query_test/test_runtime_filters.py:92: in test_bloom_wait_time assert
> duration < 60, \ E AssertionError: Query took too long (118.044356108s,
> possibly waiting for missing filters?) E assert 118.04435610771179 < 60
> Stacktrace
> query_test/test_runtime_filters.py:92: in test_bloom_wait_time
> assert duration < 60, \
> E AssertionError: Query took too long (118.044356108s, possibly waiting for
> missing filters?)
> E assert 118.04435610771179 < 60
> Standard Error
> -- executing against localhost:21000
> use functional_parquet;
> SET batch_size=0;
> SET num_nodes=0;
> SET disable_codegen_rows_threshold=0;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> -- executing against localhost:21000
> SET RUNTIME_FILTER_WAIT_TIME_MS=60;
> -- executing against localhost:21000
> SET RUNTIME_FILTER_MODE=GLOBAL;
> -- executing against localhost:21000
> SET RUNTIME_FILTER_MAX_SIZE=64K;
> -- executing against localhost:21000
> with l as (select * from tpch.lineitem UNION ALL select * from tpch.lineitem)
> select STRAIGHT_JOIN count(*) from (select * from tpch.lineitem a LIMIT 1) a
> join (select * from l LIMIT 50) b on a.l_orderkey = -b.l_orderkey;
> -- executing against localhost:21000
> SET RUNTIME_FILTER_WAIT_TIME_MS="0";
> -- executing against localhost:21000
> SET RUNTIME_FILTER_MODE="GLOBAL";
> -- executing against localhost:21000
> SET RUNTIME_FILTER_MAX_SIZE="16777216";
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-3833) Fix invalid data handling in Sequence and RCFile scanners
[ https://issues.apache.org/jira/browse/IMPALA-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-3833. --- Resolution: Fixed Fix Version/s: Impala 3.1.0 Impala 2.13.0 The above commit fixed most of the issues. I created IMPALA-7065 to track the remaining issues blocking enabling of this test. > Fix invalid data handling in Sequence and RCFile scanners > - > > Key: IMPALA-3833 > URL: https://issues.apache.org/jira/browse/IMPALA-3833 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.7.0 >Reporter: Tim Armstrong >Assignee: Pranay Singh >Priority: Critical > Labels: crash, downgraded > Fix For: Impala 2.13.0, Impala 3.1.0 > > > The fuzz testing found multiple crashes in sequence and RCFile scanners. > https://gerrit.cloudera.org/#/c/3448/ > I haven't triaged the crashes, but filing this issue to track them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7055) test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to table format AVRO is not supported")
[
https://issues.apache.org/jira/browse/IMPALA-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7055:
--
Labels: correctness flaky (was: flaky)
> test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to
> table format AVRO is not supported")
> --
>
> Key: IMPALA-7055
> URL: https://issues.apache.org/jira/browse/IMPALA-7055
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0
>Reporter: David Knupp
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: correctness, flaky
>
> This failure occurred while verifying https://gerrit.cloudera.org/c/10455/,
> but it is not related to that patch. The failing build is
> https://jenkins.impala.io/job/gerrit-verify-dryrun/2511/
> (https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2232/)
> Test appears to be (from
> [avro-writer.test|https://github.com/apache/impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/avro-writer.test]):
> {noformat}
> QUERY
> SET ALLOW_UNSUPPORTED_FORMATS=0;
> insert into __avro_write select 1, "b", 2.2;
> CATCH
> Writing to table format AVRO is not supported. Use query option
> ALLOW_UNSUPPORTED_FORMATS
> {noformat}
> Error output:
> {noformat}
> 01:50:18 ] FAIL
> query_test/test_compressed_formats.py::TestTableWriters::()::test_avro_writer[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> 01:50:18 ] === FAILURES
> ===
> 01:50:18 ] TestTableWriters.test_avro_writer[exec_option: {'batch_size': 0,
> 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen':
> False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> 01:50:18 ] [gw9] linux2 -- Python 2.7.12
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 01:50:18 ] query_test/test_compressed_formats.py:189: in test_avro_writer
> 01:50:18 ] self.run_test_case('QueryTest/avro-writer', vector)
> 01:50:18 ] common/impala_test_suite.py:420: in run_test_case
> 01:50:18 ] assert False, "Expected exception: %s" % expected_str
> 01:50:18 ] E AssertionError: Expected exception: Writing to table format
> AVRO is not supported. Use query option ALLOW_UNSUPPORTED_FORMATS
> 01:50:18 ] Captured stderr setup
> -
> 01:50:18 ] -- connecting to: localhost:21000
> 01:50:18 ] - Captured stderr call
> -
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] use functional;
> 01:50:18 ]
> 01:50:18 ] SET batch_size=0;
> 01:50:18 ] SET num_nodes=0;
> 01:50:18 ] SET disable_codegen_rows_threshold=5000;
> 01:50:18 ] SET disable_codegen=False;
> 01:50:18 ] SET abort_on_error=1;
> 01:50:18 ] SET exec_single_node_rows_threshold=0;
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] drop table if exists __avro_write;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=NONE;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] create table __avro_write (i int, s string, d double)
> 01:50:18 ] stored as AVRO
> 01:50:18 ] TBLPROPERTIES ('avro.schema.literal'='{
> 01:50:18 ] "name": "my_record",
> 01:50:18 ] "type": "record",
> 01:50:18 ] "fields": [
> 01:50:18 ] {"name":"i", "type":["int", "null"]},
> 01:50:18 ] {"name":"s", "type":["string", "null"]},
> 01:50:18 ] {"name":"d", "type":["double", "null"]}]}');
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC="";
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=NONE;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS=1;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] insert into __avro_write select 0, "a", 1.1;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC="";
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS="0";
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=SNAPPY;
> 01:50:18 ]
> 01:50:18 ] -- executing against
[jira] [Assigned] (IMPALA-7055) test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to table format AVRO is not supported")
[
https://issues.apache.org/jira/browse/IMPALA-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-7055:
-
Assignee: Tim Armstrong
> test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to
> table format AVRO is not supported")
> --
>
> Key: IMPALA-7055
> URL: https://issues.apache.org/jira/browse/IMPALA-7055
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0
>Reporter: David Knupp
>Assignee: Tim Armstrong
>Priority: Major
> Labels: flaky
>
> This failure occurred while verifying https://gerrit.cloudera.org/c/10455/,
> but it is not related to that patch. The failing build is
> https://jenkins.impala.io/job/gerrit-verify-dryrun/2511/
> (https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2232/)
> Test appears to be (from
> [avro-writer.test|https://github.com/apache/impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/avro-writer.test]):
> {noformat}
> QUERY
> SET ALLOW_UNSUPPORTED_FORMATS=0;
> insert into __avro_write select 1, "b", 2.2;
> CATCH
> Writing to table format AVRO is not supported. Use query option
> ALLOW_UNSUPPORTED_FORMATS
> {noformat}
> Error output:
> {noformat}
> 01:50:18 ] FAIL
> query_test/test_compressed_formats.py::TestTableWriters::()::test_avro_writer[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> 01:50:18 ] === FAILURES
> ===
> 01:50:18 ] TestTableWriters.test_avro_writer[exec_option: {'batch_size': 0,
> 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen':
> False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> 01:50:18 ] [gw9] linux2 -- Python 2.7.12
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 01:50:18 ] query_test/test_compressed_formats.py:189: in test_avro_writer
> 01:50:18 ] self.run_test_case('QueryTest/avro-writer', vector)
> 01:50:18 ] common/impala_test_suite.py:420: in run_test_case
> 01:50:18 ] assert False, "Expected exception: %s" % expected_str
> 01:50:18 ] E AssertionError: Expected exception: Writing to table format
> AVRO is not supported. Use query option ALLOW_UNSUPPORTED_FORMATS
> 01:50:18 ] Captured stderr setup
> -
> 01:50:18 ] -- connecting to: localhost:21000
> 01:50:18 ] - Captured stderr call
> -
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] use functional;
> 01:50:18 ]
> 01:50:18 ] SET batch_size=0;
> 01:50:18 ] SET num_nodes=0;
> 01:50:18 ] SET disable_codegen_rows_threshold=5000;
> 01:50:18 ] SET disable_codegen=False;
> 01:50:18 ] SET abort_on_error=1;
> 01:50:18 ] SET exec_single_node_rows_threshold=0;
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] drop table if exists __avro_write;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=NONE;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] create table __avro_write (i int, s string, d double)
> 01:50:18 ] stored as AVRO
> 01:50:18 ] TBLPROPERTIES ('avro.schema.literal'='{
> 01:50:18 ] "name": "my_record",
> 01:50:18 ] "type": "record",
> 01:50:18 ] "fields": [
> 01:50:18 ] {"name":"i", "type":["int", "null"]},
> 01:50:18 ] {"name":"s", "type":["string", "null"]},
> 01:50:18 ] {"name":"d", "type":["double", "null"]}]}');
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC="";
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=NONE;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS=1;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] insert into __avro_write select 0, "a", 1.1;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC="";
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS="0";
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=SNAPPY;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
>
[jira] [Created] (IMPALA-7065) Fix remaining invalid data handling bugs in RC and sequence file
Tim Armstrong created IMPALA-7065: - Summary: Fix remaining invalid data handling bugs in RC and sequence file Key: IMPALA-7065 URL: https://issues.apache.org/jira/browse/IMPALA-7065 Project: IMPALA Issue Type: Bug Components: Backend Affects Versions: Impala 2.13.0, Impala 3.1.0 Reporter: Tim Armstrong IMPALA-7058 revealed some problems that weren't handled by the original IMPALA-3833 patch. We should fix the remaining cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7055) test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to table format AVRO is not supported")
[
https://issues.apache.org/jira/browse/IMPALA-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7055:
--
Description:
This failure occurred while verifying https://gerrit.cloudera.org/c/10455/, but
it is not related to that patch. The failing build is
https://jenkins.impala.io/job/gerrit-verify-dryrun/2511/
(https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2232/)
Test appears to be (from
[avro-writer.test|https://github.com/apache/impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/avro-writer.test]):
{noformat}
QUERY
SET ALLOW_UNSUPPORTED_FORMATS=0;
insert into __avro_write select 1, "b", 2.2;
CATCH
Writing to table format AVRO is not supported. Use query option
ALLOW_UNSUPPORTED_FORMATS
{noformat}
Error output:
{noformat}
01:50:18 ] FAIL
query_test/test_compressed_formats.py::TestTableWriters::()::test_avro_writer[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: text/none]
01:50:18 ] === FAILURES
===
01:50:18 ] TestTableWriters.test_avro_writer[exec_option: {'batch_size': 0,
'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen':
False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: text/none]
01:50:18 ] [gw9] linux2 -- Python 2.7.12
/home/ubuntu/Impala/bin/../infra/python/env/bin/python
01:50:18 ] query_test/test_compressed_formats.py:189: in test_avro_writer
01:50:18 ] self.run_test_case('QueryTest/avro-writer', vector)
01:50:18 ] common/impala_test_suite.py:420: in run_test_case
01:50:18 ] assert False, "Expected exception: %s" % expected_str
01:50:18 ] E AssertionError: Expected exception: Writing to table format AVRO
is not supported. Use query option ALLOW_UNSUPPORTED_FORMATS
01:50:18 ] Captured stderr setup
-
01:50:18 ] -- connecting to: localhost:21000
01:50:18 ] - Captured stderr call
-
01:50:18 ] -- executing against localhost:21000
01:50:18 ] use functional;
01:50:18 ]
01:50:18 ] SET batch_size=0;
01:50:18 ] SET num_nodes=0;
01:50:18 ] SET disable_codegen_rows_threshold=5000;
01:50:18 ] SET disable_codegen=False;
01:50:18 ] SET abort_on_error=1;
01:50:18 ] SET exec_single_node_rows_threshold=0;
01:50:18 ] -- executing against localhost:21000
01:50:18 ] drop table if exists __avro_write;
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ] SET COMPRESSION_CODEC=NONE;
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ]
01:50:18 ] create table __avro_write (i int, s string, d double)
01:50:18 ] stored as AVRO
01:50:18 ] TBLPROPERTIES ('avro.schema.literal'='{
01:50:18 ] "name": "my_record",
01:50:18 ] "type": "record",
01:50:18 ] "fields": [
01:50:18 ] {"name":"i", "type":["int", "null"]},
01:50:18 ] {"name":"s", "type":["string", "null"]},
01:50:18 ] {"name":"d", "type":["double", "null"]}]}');
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ] SET COMPRESSION_CODEC="";
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ] SET COMPRESSION_CODEC=NONE;
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ]
01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS=1;
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ]
01:50:18 ] insert into __avro_write select 0, "a", 1.1;
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ] SET COMPRESSION_CODEC="";
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS="0";
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ] SET COMPRESSION_CODEC=SNAPPY;
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ]
01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS=1;
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ]
01:50:18 ] insert into __avro_write select 1, "b", 2.2;
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ] SET COMPRESSION_CODEC="";
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS="0";
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ] select * from __avro_write;
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS=0;
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ]
01:50:18 ] insert into __avro_write select 1, "b", 2.2;
01:50:18 ]
01:50:18 ] -- executing against localhost:21000
01:50:18 ] SET
[jira] [Commented] (IMPALA-7055) test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to table format AVRO is not supported")
[
https://issues.apache.org/jira/browse/IMPALA-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487993#comment-16487993
]
Tim Armstrong commented on IMPALA-7055:
---
Here are the log lines with the query id. It ran on the coordinator only.
{noformat}
I0519 00:25:19.680714 38345 admission-controller.cc:508] Schedule for
id=4d4d0db68525d9ff:3bea3eee in pool_name=default-pool
cluster_mem_needed=10.00 MB PoolConfig: max_requests=-1 max_queued=200
max_mem=-1.00 B
I0519 00:25:19.680757 38345 admission-controller.cc:529] Admitted query
id=4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.680784 38345 coordinator.cc:90] Exec()
query_id=4d4d0db68525d9ff:3bea3eee stmt=insert into __avro_write select
1, "b", 2.2
I0519 00:25:19.681054 38345 coordinator.cc:335] starting execution on 1
backends for query_id=4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.681506 124530 impala-internal-service.cc:44]
ExecQueryFInstances(): query_id=4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.681530 124530 query-exec-mgr.cc:46] StartQueryFInstances()
query_id=4d4d0db68525d9ff:3bea3eee coord=ip-172-31-9-142:22000
I0519 00:25:19.681543 124530 query-state.cc:178] Buffer pool limit for
4d4d0db68525d9ff:3bea3eee: 10307921510
I0519 00:25:19.681646 124530 initial-reservations.cc:60] Successfully claimed
initial reservations (0) for query 4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.693480 41165 query-state.cc:300] StartFInstances():
query_id=4d4d0db68525d9ff:3bea3eee #instances=1
I0519 00:25:19.693578 41165 query-state.cc:313] descriptor table for
query=4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.693596 38345 coordinator.cc:348] started execution on 1 backends
for query_id=4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.701475 41169 query-state.cc:395] Executing instance.
instance_id=4d4d0db68525d9ff:3bea3eee fragment_idx=0
per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=11
I0519 00:25:19.702011 41170 coordinator.cc:585] Coordinator waiting for
backends to finish, 1 remaining. query_id=4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.713778 41165 query-exec-mgr.cc:155] ReleaseQueryState():
query_id=4d4d0db68525d9ff:3bea3eee refcnt=3
I0519 00:25:19.915730 41170 coordinator.cc:554] Finalizing query:
4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.915823 41170 coordinator.cc:576] Removing staging directory:
hdfs://localhost:20500/test-warehouse/functional.db/__avro_write/_impala_insert_staging/4d4d0db68525d9ff_3bea3eee/
I0519 00:25:19.916484 41170 coordinator.cc:454] ExecState: query
id=4d4d0db68525d9ff:3bea3eee execution completed
I0519 00:25:19.916503 41170 coordinator.cc:763] Release admission control
resources for query_id=4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.917029 41170 client-request-state.cc:928] No partitions altered,
not updating metastore (query id: 4d4d0db68525d9ff:3bea3eee)
I0519 00:25:19.915721 76770 coordinator.cc:684] Backend completed:
host=ip-172-31-9-142:22000 remaining=0
query_id=4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.917754 76770 coordinator.cc:495] ExecState: query
id=4d4d0db68525d9ff:3bea3eee
finstance=4d4d0db68525d9ff:3bea3eee on host=ip-172-31-9-142:22000
(RETURNED_RESULTS -> RETURNED_RESULTS) status=Writing to table format AVRO is
not supported. Use query option ALLOW_UNSUPPORTED_FORMATS to override.
I0519 00:25:19.918164 41169 query-state.cc:416] Cancel:
query_id=4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.918174 41169 krpc-data-stream-mgr.cc:324] cancelling all streams
for fragment_instance_id=4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.918341 41169 query-state.cc:403] Instance completed.
instance_id=4d4d0db68525d9ff:3bea3eee #in-flight=8 status=GENERAL:
Writing to table format AVRO is not supported. Use query option
ALLOW_UNSUPPORTED_FORMATS to override.
I0519 00:25:19.918355 41169 query-state.cc:416] Cancel:
query_id=4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.918377 41169 query-exec-mgr.cc:155] ReleaseQueryState():
query_id=4d4d0db68525d9ff:3bea3eee refcnt=2
I0519 00:25:19.960108 38345 impala-beeswax-server.cc:355] CloseInsert():
query_id=4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.960124 38345 impala-server.cc:1010] UnregisterQuery():
query_id=4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.960130 38345 impala-server.cc:1097] Cancel():
query_id=4d4d0db68525d9ff:3bea3eee
I0519 00:25:19.962421 38345 query-exec-mgr.cc:155] ReleaseQueryState():
query_id=4d4d0db68525d9ff:3bea3eee refcnt=1
{noformat}
> test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to
> table format AVRO is not supported")
> --
>
> Key: IMPALA-7055
> URL:
[jira] [Created] (IMPALA-7067) sleep(100000) command from test_shell_commandline.py can hang around and cause test_metrics_are_zero to fail
Tim Armstrong created IMPALA-7067:
-
Summary: sleep(10) command from test_shell_commandline.py can
hang around and cause test_metrics_are_zero to fail
Key: IMPALA-7067
URL: https://issues.apache.org/jira/browse/IMPALA-7067
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong
{noformat}
03:25:47 [gw6] PASSED
shell/test_shell_commandline.py::TestImpalaShell::test_cancellation
...
03:27:01 verifiers/test_verify_metrics.py:34: in test_metrics_are_zero
03:27:01 verifier.verify_metrics_are_zero()
03:27:01 verifiers/metric_verifier.py:47: in verify_metrics_are_zero
03:27:01 self.wait_for_metric(metric, 0, timeout)
03:27:01 verifiers/metric_verifier.py:62: in wait_for_metric
03:27:01 self.impalad_service.wait_for_metric_value(metric_name,
expected_value, timeout)
03:27:01 common/impala_service.py:135: in wait_for_metric_value
03:27:01 json.dumps(self.read_debug_webpage('rpcz?json')))
03:27:01 E AssertionError: Metric value impala-server.mem-pool.total-bytes
did not reach value 0 in 60s
{noformat}
I used the json dump from memz and the logs to trace it back to the
sleep(10) query hanging around
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7067) sleep(100000) command from test_shell_commandline.py can hang around and cause test_metrics_are_zero to fail
[
https://issues.apache.org/jira/browse/IMPALA-7067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7067:
--
Target Version: Impala 2.13.0, Impala 3.1.0 (was: Impala 3.1.0)
> sleep(10) command from test_shell_commandline.py can hang around and
> cause test_metrics_are_zero to fail
>
>
> Key: IMPALA-7067
> URL: https://issues.apache.org/jira/browse/IMPALA-7067
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: flaky
>
> {noformat}
> 03:25:47 [gw6] PASSED
> shell/test_shell_commandline.py::TestImpalaShell::test_cancellation
> ...
> 03:27:01 verifiers/test_verify_metrics.py:34: in test_metrics_are_zero
> 03:27:01 verifier.verify_metrics_are_zero()
> 03:27:01 verifiers/metric_verifier.py:47: in verify_metrics_are_zero
> 03:27:01 self.wait_for_metric(metric, 0, timeout)
> 03:27:01 verifiers/metric_verifier.py:62: in wait_for_metric
> 03:27:01 self.impalad_service.wait_for_metric_value(metric_name,
> expected_value, timeout)
> 03:27:01 common/impala_service.py:135: in wait_for_metric_value
> 03:27:01 json.dumps(self.read_debug_webpage('rpcz?json')))
> 03:27:01 E AssertionError: Metric value impala-server.mem-pool.total-bytes
> did not reach value 0 in 60s
> {noformat}
> I used the json dump from memz and the logs to trace it back to the
> sleep(10) query hanging around
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-5837) TestImpalaShell.test_cancellation fails on ASAN build: Could not execute command: select sleep(10000)
[
https://issues.apache.org/jira/browse/IMPALA-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-5837.
---
Resolution: Cannot Reproduce
> TestImpalaShell.test_cancellation fails on ASAN build: Could not execute
> command: select sleep(1)
> --
>
> Key: IMPALA-5837
> URL: https://issues.apache.org/jira/browse/IMPALA-5837
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 2.10.0
>Reporter: David Knupp
>Priority: Minor
> Labels: flaky
>
> This test has been marked as being flaky on the ASAN build in the past
> (IMPALA-4046).
> This failure is different from that, so I'm not reopening that JIRA -- just
> noting that it's the same test and the same build flavor.
> {noformat}
> 02:21:28 shell/test_shell_commandline.py:324: in test_cancellation
> 02:21:28 assert "Cancelling Query" in result.stderr, result.stderr
> 02:21:28 E AssertionError: Starting Impala Shell without Kerberos
> authentication
> 02:21:28 E Connected to localhost:21000
> 02:21:28 E Server version: impalad version 2.10.0-SNAPSHOT DEBUG (build
> c67b198a19eda12c99905c2b9db30cc86f6e332d)
> 02:21:28 E Query: select sleep(1)
> 02:21:28 E Query submitted at: 2017-08-24 01:58:21 (Coordinator:
> http://impala-boost-static-burst-slave-0228.vpc.cloudera.com:25000)
> 02:21:28 E Could not execute command: select sleep(1)
> 02:21:28 E
> 02:21:28 E assert 'Cancelling Query' in 'Starting Impala Shell without
> Kerberos authentication\nConnected to localhost:21000\nServer version:
> impalad version ...
> http://impala-boost-static-burst-slave-0228.vpc.cloudera.com:25000)\nCould
> not execute command: select sleep(1)\n'
> 02:21:28 E+ where 'Starting Impala Shell without Kerberos
> authentication\nConnected to localhost:21000\nServer version: impalad version
> ...
> http://impala-boost-static-burst-slave-0228.vpc.cloudera.com:25000)\nCould
> not execute command: select sleep(1)\n' =
> .stderr
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7073) Failed test: query_test.test_scanners.TestScannerReservation.test_scanners
[
https://issues.apache.org/jira/browse/IMPALA-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7073:
--
Description:
Possibly flaky test:
{code:java}
Stacktrace
query_test/test_scanners.py:1064: in test_scanners
self.run_test_case('QueryTest/scanner-reservation', vector)
common/impala_test_suite.py:451: in run_test_case
verify_runtime_profile(test_section['RUNTIME_PROFILE'],
result.runtime_profile)
common/test_result_verifier.py:590: in verify_runtime_profile
actual))
E AssertionError: Did not find matches for lines in runtime profile:
E EXPECTED LINES:
E row_regex:.*InitialRangeActualReservation.*Avg: 4.00 MB.*{code}
{noformat}
11:27:13 -- executing against localhost:21000
11:27:13 set debug_action="-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0";
11:27:13
11:27:13 -- executing against localhost:21000
11:27:13
11:27:13 select count(*)
11:27:13 from tpch.customer;
11:27:13
11:27:13 -- executing against localhost:21000
11:27:13 SET DEBUG_ACTION="";
11:27:13
11:27:13 -- executing against localhost:21000
11:27:13 select min(l_comment)
11:27:13 from tpch_parquet.lineitem;
{noformat}
{noformat}
11:27:13 E - SpilledPartitions: 0 (0)
11:27:13 E HDFS_SCAN_NODE (id=0):(Total: 2s295ms, non-child: 2s295ms,
% non-child: 100.00%)
11:27:13 E Hdfs split stats (:<# splits>/): -1:8/193.99 MB
11:27:13 E ExecOption: PARQUET Codegen Enabled, Codegen enabled: 8
out of 8
11:27:13 E Hdfs Read Thread Concurrency Bucket: 0:80% 1:20% 2:0%
3:0% 4:0% 5:0% 6:0%
11:27:13 E File Formats: PARQUET/NONE:5 PARQUET/SNAPPY:3
11:27:13 E BytesRead(500.000ms): 0, 21.31 MB, 21.60 MB, 47.94 MB,
56.23 MB, 74.46 MB
11:27:13 E - FooterProcessingTime: (Avg: 3.624ms ; Min: 999.979us
; Max: 9.999ms ; Number of samples: 8)
11:27:13 E - InitialRangeActualReservation: (Avg: 21.50 MB
(22544384) ; Min: 4.00 MB (4194304) ; Max: 24.00 MB (25165824) ; Number of
samples: 8)
11:27:13 E - InitialRangeIdealReservation: (Avg: 128.00 KB
(131072) ; Min: 128.00 KB (131072) ; Max: 128.00 KB (131072) ; Number of
samples: 8)
11:27:13 E - ParquetRowGroupActualReservation: (Avg: 24.00 MB
(25165824) ; Min: 24.00 MB (25165824) ; Max: 24.00 MB (25165824) ; Number of
samples: 3)
11:27:13 E - ParquetRowGroupIdealReservation: (Avg: 24.00 MB
(25165824) ; Min: 24.00 MB (25165824) ; Max: 24.00 MB (25165824) ; Number of
samples: 3)
11:27:13 E - AverageHdfsReadThreadConcurrency: 0.20
11:27:13 E - AverageScannerThreadConcurrency: 1.00
11:27:13 E - BytesRead: 74.55 MB (78175787)
11:27:13 E - BytesReadDataNodeCache: 0
11:27:13 E - BytesReadLocal: 0
11:27:13 E - BytesReadRemoteUnexpected: 0
11:27:13 E - BytesReadShortCircuit: 0
11:27:13 E - CachedFileHandlesHitCount: 0 (0)
11:27:13 E - CachedFileHandlesMissCount: 11 (11)
11:27:13 E - CollectionItemsRead: 0 (0)
11:27:13 E - DecompressionTime: 345.992ms
11:27:13 E - MaxCompressedTextFileLength: 0
11:27:13 E - NumColumns: 1 (1)
11:27:13 E - NumDictFilteredRowGroups: 0 (0)
11:27:13 E - NumDisksAccessed: 2 (2)
11:27:13 E - NumRowGroups: 3 (3)
11:27:13 E - NumScannerThreadReservationsDenied: 0 (0)
11:27:13 E - NumScannerThreadsStarted: 1 (1)
11:27:13 E - NumScannersWithNoReads: 5 (5)
11:27:13 E - NumStatsFilteredRowGroups: 0 (0)
11:27:13 E - PeakMemoryUsage: 26.40 MB (27684346)
11:27:13 E - PerReadThreadRawHdfsThroughput: 140.94 MB/sec
11:27:13 E - RemoteScanRanges: 0 (0)
11:27:13 E - RowBatchQueueGetWaitTime: 2s179ms
11:27:13 E - RowBatchQueuePutWaitTime: 0.000ns
11:27:13 E - RowsRead: 6.00M (6001215)
11:27:13 E - RowsReturned: 6.00M (6001215)
11:27:13 E - RowsReturnedRate: 2.61 M/sec
11:27:13 E - ScanRangesComplete: 8 (8)
11:27:13 E - ScannerThreadsInvoluntaryContextSwitches: 6.62K (6622)
11:27:13 E - ScannerThreadsTotalWallClockTime: 2s509ms
11:27:13 E- MaterializeTupleTime(*): 980.979ms
11:27:13 E- ScannerThreadsSysTime: 181.972ms
11:27:13 E- ScannerThreadsUserTime: 965.853ms
11:27:13 E - ScannerThreadsVoluntaryContextSwitches: 19 (19)
11:27:13 E - TotalRawHdfsOpenFileTime(*): 21.999ms
11:27:13 E - TotalRawHdfsReadTime(*): 528.988ms
11:27:13 E - TotalReadThroughput: 24.82 MB/sec
11:27:13 E Buffer pool:
11:27:13 E- AllocTime: 0.000ns
11:27:13 E- CumulativeAllocationBytes: 73.00 MB (76546048)
11:27:13 E-
[jira] [Commented] (IMPALA-7067) sleep(100000) command from test_shell_commandline.py can hang around and cause test_metrics_are_zero to fail
[
https://issues.apache.org/jira/browse/IMPALA-7067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489329#comment-16489329
]
Tim Armstrong commented on IMPALA-7067:
---
It's a test bug - the fragment thread is stuck in the sleep() call so can't
clean itself up.
> sleep(10) command from test_shell_commandline.py can hang around and
> cause test_metrics_are_zero to fail
>
>
> Key: IMPALA-7067
> URL: https://issues.apache.org/jira/browse/IMPALA-7067
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: flaky
>
> {noformat}
> 03:25:47 [gw6] PASSED
> shell/test_shell_commandline.py::TestImpalaShell::test_cancellation
> ...
> 03:27:01 verifiers/test_verify_metrics.py:34: in test_metrics_are_zero
> 03:27:01 verifier.verify_metrics_are_zero()
> 03:27:01 verifiers/metric_verifier.py:47: in verify_metrics_are_zero
> 03:27:01 self.wait_for_metric(metric, 0, timeout)
> 03:27:01 verifiers/metric_verifier.py:62: in wait_for_metric
> 03:27:01 self.impalad_service.wait_for_metric_value(metric_name,
> expected_value, timeout)
> 03:27:01 common/impala_service.py:135: in wait_for_metric_value
> 03:27:01 json.dumps(self.read_debug_webpage('rpcz?json')))
> 03:27:01 E AssertionError: Metric value impala-server.mem-pool.total-bytes
> did not reach value 0 in 60s
> {noformat}
> I used the json dump from memz and the logs to trace it back to the
> sleep(10) query hanging around
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7069) Java UDF tests can trigger a crash in java.util.concurrent.ConcurrentHashMap.putVal()
Tim Armstrong created IMPALA-7069:
-
Summary: Java UDF tests can trigger a crash in
java.util.concurrent.ConcurrentHashMap.putVal()
Key: IMPALA-7069
URL: https://issues.apache.org/jira/browse/IMPALA-7069
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.1.0
Reporter: Tim Armstrong
Assignee: Vuk Ercegovac
Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
hs_err_pid8975.log, hs_err_pid9694.log
I hit this crash on a GVO, but was able to reproduce it on master on my desktop.
Repro steps:
{code}
git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
./buildall.sh -debug -noclean -notests -skiptests -ninja
start-impala-cluster.py
while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar' -n4
--verbose; do date; done
{code}
I generally hit the crash within a hour of looping the test.
{noformat}
Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
space=8175k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x8a8107]
V [libjvm.so+0x96cf5f]
v ~RuntimeStub::_complete_monitor_locking_Java
J 2758 C2
java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
(362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
J 2311 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
(122 bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
J 3953 C2
java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
(40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
J 2987 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;
(7 bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
v ~StubRoutines::call_stub
V [libjvm.so+0x6648eb]
V [libjvm.so+0x661ec4]
V [libjvm.so+0x662523]
V [libjvm.so+0x9e398d]
V [libjvm.so+0x9e2326]
V [libjvm.so+0x9e2b50]
V [libjvm.so+0x42c099]
V [libjvm.so+0x9dc786]
V [libjvm.so+0x6a5edf]
V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
V [libjvm.so+0xa31ea5]
V [libjvm.so+0xa37ea7]
J 4842
sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
(0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
v ~StubRoutines::call_stub
V [libjvm.so+0x6648eb]
V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
J 1035
java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
(0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
J 20421 C2
sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
(762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
J 4163 C2
sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes) @
0x7fa0c711c638 [0x7fa0c711c400+0x238]
v ~StubRoutines::call_stub
V [libjvm.so+0x6648eb]
V [libjvm.so+0x6822d7]
V [libjvm.so+0x6862c9]
C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*, _jclass*,
_jmethodID*, jvalue const*)+0x40
C [impalad+0x29fe4ff]
impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
const*) const+0x44b
C [impalad+0x29ffde9]
impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
impala::TupleRow const*) const+0xbb
C [impalad+0x2a0948a]
impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
impala::TupleRow const*)+0x14c
C [impalad+0x2a48eb1]
impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
impala::TupleRow const*) const+0x9d
C [impalad+0x2a4abba] impala_udf::BooleanVal
impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
impala::TupleRow const*) const+0x18c
C [impalad+0x2a4907d]
impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*,
impala::TupleRow const*) const+0x179
C [impalad+0x2a09c7f]
impala::ScalarExprEvaluator::GetBooleanVal(impala::TupleRow*)+0x37
C [impalad+0x1b70efb]
impala::ExecNode::EvalPredicate(impala::ScalarExprEvaluator*,
impala::TupleRow*)+0x23
C [impalad+0x1b70efb]
impala::ExecNode::EvalPredicate(impala::ScalarExprEvaluator*,
impala::TupleRow*)+0x23
C [impalad+0x1b6fdf0]
impala::ExecNode::EvalConjuncts(impala::ScalarExprEvaluator* const*, int,
impala::TupleRow*)+0x42
C [impalad+0x1bb60e3]
impala::HdfsScanner::EvalConjuncts(impala::TupleRow*)+0x4d
C [impalad+0x1bb08fd]
impala::HdfsScanner::WriteCompleteTuple(impala::MemPool*,
[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489358#comment-16489358
]
Tim Armstrong commented on IMPALA-7069:
---
Assigned to Vuk since he's looked at the UDF loading most recently. I attached
the hs_err_pid files from my repro attempts.
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Vuk Ercegovac
>Priority: Critical
> Labels: crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C [impalad+0x2a4907d]
> impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x179
> C [impalad+0x2a09c7f]
>
[jira] [Updated] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7069:
--
Summary: Java UDF tests can trigger a crash in Java ClassLoader (was: Java
UDF tests can trigger a crash in
java.util.concurrent.ConcurrentHashMap.putVal())
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Vuk Ercegovac
>Priority: Critical
> Labels: crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C [impalad+0x2a4907d]
> impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x179
> C [impalad+0x2a09c7f]
>
[jira] [Commented] (IMPALA-7069) Java UDF tests can trigger a crash in Java ClassLoader
[
https://issues.apache.org/jira/browse/IMPALA-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489522#comment-16489522
]
Tim Armstrong commented on IMPALA-7069:
---
I saw it on a commit branched off fd7e7c93c5d2ae153784548a2f83f423e85dda43 .
AFAIK we didn't see it before yesterday.
> Java UDF tests can trigger a crash in Java ClassLoader
> --
>
> Key: IMPALA-7069
> URL: https://issues.apache.org/jira/browse/IMPALA-7069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Vuk Ercegovac
>Priority: Critical
> Labels: crash, flaky
> Attachments: hs_err_pid22764.log, hs_err_pid29246.log,
> hs_err_pid8975.log, hs_err_pid9694.log
>
>
> I hit this crash on a GVO, but was able to reproduce it on master on my
> desktop.
> Repro steps:
> {code}
> git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
> ./buildall.sh -debug -noclean -notests -skiptests -ninja
> start-impala-cluster.py
> while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar'
> -n4 --verbose; do date; done
> {code}
> I generally hit the crash within a hour of looping the test.
> {noformat}
> Stack: [0x7fa04791f000,0x7fa04812], sp=0x7fa04811aff0, free
> space=8175k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x8a8107]
> V [libjvm.so+0x96cf5f]
> v ~RuntimeStub::_complete_monitor_locking_Java
> J 2758 C2
> java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
> (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
> J 2311 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
> bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
> J 3953 C2
> java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
> (40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
> J 2987 C2
> java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7
> bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x661ec4]
> V [libjvm.so+0x662523]
> V [libjvm.so+0x9e398d]
> V [libjvm.so+0x9e2326]
> V [libjvm.so+0x9e2b50]
> V [libjvm.so+0x42c099]
> V [libjvm.so+0x9dc786]
> V [libjvm.so+0x6a5edf]
> V [libjvm.so+0x6a70cb] JVM_DefineClass+0xbb
> V [libjvm.so+0xa31ea5]
> V [libjvm.so+0xa37ea7]
> J 4842
> sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
> (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
> J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5
> bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6b5949] JVM_DoPrivileged+0x429
> J 1035
> java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
> (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
> J 20421 C2
> sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
> (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
> J 4163 C2
> sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
> J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes)
> @ 0x7fa0c711c638 [0x7fa0c711c400+0x238]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x6648eb]
> V [libjvm.so+0x6822d7]
> V [libjvm.so+0x6862c9]
> C [impalad+0x2a004fa] JNIEnv_::CallNonvirtualVoidMethodA(_jobject*,
> _jclass*, _jmethodID*, jvalue const*)+0x40
> C [impalad+0x29fe4ff]
> impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow
> const*) const+0x44b
> C [impalad+0x29ffde9]
> impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0xbb
> C [impalad+0x2a0948a]
> impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
> impala::TupleRow const*)+0x14c
> C [impalad+0x2a48eb1]
> impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x9d
> C [impalad+0x2a4abba] impala_udf::BooleanVal
> impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x18c
> C [impalad+0x2a4907d]
> impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*,
> impala::TupleRow const*) const+0x179
> C [impalad+0x2a09c7f]
>
[jira] [Resolved] (IMPALA-7055) test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to table format AVRO is not supported")
[
https://issues.apache.org/jira/browse/IMPALA-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-7055.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
Impala 2.13.0
> test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to
> table format AVRO is not supported")
> --
>
> Key: IMPALA-7055
> URL: https://issues.apache.org/jira/browse/IMPALA-7055
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0
>Reporter: David Knupp
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: correctness, flaky
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> This failure occurred while verifying https://gerrit.cloudera.org/c/10455/,
> but it is not related to that patch. The failing build is
> https://jenkins.impala.io/job/gerrit-verify-dryrun/2511/
> (https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2232/)
> Test appears to be (from
> [avro-writer.test|https://github.com/apache/impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/avro-writer.test]):
> {noformat}
> QUERY
> SET ALLOW_UNSUPPORTED_FORMATS=0;
> insert into __avro_write select 1, "b", 2.2;
> CATCH
> Writing to table format AVRO is not supported. Use query option
> ALLOW_UNSUPPORTED_FORMATS
> {noformat}
> Error output:
> {noformat}
> 01:50:18 ] FAIL
> query_test/test_compressed_formats.py::TestTableWriters::()::test_avro_writer[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> 01:50:18 ] === FAILURES
> ===
> 01:50:18 ] TestTableWriters.test_avro_writer[exec_option: {'batch_size': 0,
> 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen':
> False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> 01:50:18 ] [gw9] linux2 -- Python 2.7.12
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 01:50:18 ] query_test/test_compressed_formats.py:189: in test_avro_writer
> 01:50:18 ] self.run_test_case('QueryTest/avro-writer', vector)
> 01:50:18 ] common/impala_test_suite.py:420: in run_test_case
> 01:50:18 ] assert False, "Expected exception: %s" % expected_str
> 01:50:18 ] E AssertionError: Expected exception: Writing to table format
> AVRO is not supported. Use query option ALLOW_UNSUPPORTED_FORMATS
> 01:50:18 ] Captured stderr setup
> -
> 01:50:18 ] -- connecting to: localhost:21000
> 01:50:18 ] - Captured stderr call
> -
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] use functional;
> 01:50:18 ]
> 01:50:18 ] SET batch_size=0;
> 01:50:18 ] SET num_nodes=0;
> 01:50:18 ] SET disable_codegen_rows_threshold=5000;
> 01:50:18 ] SET disable_codegen=False;
> 01:50:18 ] SET abort_on_error=1;
> 01:50:18 ] SET exec_single_node_rows_threshold=0;
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] drop table if exists __avro_write;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=NONE;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] create table __avro_write (i int, s string, d double)
> 01:50:18 ] stored as AVRO
> 01:50:18 ] TBLPROPERTIES ('avro.schema.literal'='{
> 01:50:18 ] "name": "my_record",
> 01:50:18 ] "type": "record",
> 01:50:18 ] "fields": [
> 01:50:18 ] {"name":"i", "type":["int", "null"]},
> 01:50:18 ] {"name":"s", "type":["string", "null"]},
> 01:50:18 ] {"name":"d", "type":["double", "null"]}]}');
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC="";
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=NONE;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS=1;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ]
> 01:50:18 ] insert into __avro_write select 0, "a", 1.1;
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC="";
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS="0";
> 01:50:18 ]
> 01:50:18 ] -- executing against localhost:21000
>
[jira] [Commented] (IMPALA-7012) NullPointerException with CTAS query
[
https://issues.apache.org/jira/browse/IMPALA-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489849#comment-16489849
]
Tim Armstrong commented on IMPALA-7012:
---
I'm not sure, I was probably doing something wrong here, but it shouldn't
generate a NPE regardless
> NullPointerException with CTAS query
>
>
> Key: IMPALA-7012
> URL: https://issues.apache.org/jira/browse/IMPALA-7012
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Tianyi Wang
>Priority: Critical
>
> {noformat}
> [localhost:21000] default> create table alltypesinsert partitioned by (year,
> month) stored as parquet as /* +noclustered */select at1.id, at1.bool_col,
> at1.tinyint_col, at1.smallint_col, at1.int_col, at1.bigint_col,
>
> at1.float_col, at1.double_col, at1.date_string_col, at1.string_col,
> at1.timestamp_col,
> at1.year, at2.id as month
> from functional.alltypes at1, functional.alltypes at2;
> Query: create table alltypesinsert partitioned by (year, month) stored as
> parquet as /* +noclustered */
> select at1.id, at1.bool_col, at1.tinyint_col, at1.smallint_col, at1.int_col,
> at1.bigint_col,
> at1.float_col, at1.double_col, at1.date_string_col, at1.string_col,
> at1.timestamp_col,
> at1.year, at2.id as month
> from functional.alltypes at1, functional.alltypes at2
> Query submitted at: 2018-05-10 13:46:02 (Coordinator:
> http://tarmstrong-box:25000)
> ERROR: NullPointerException: null
> {noformat}
> {noformat}
> I0510 13:46:02.977249 4238 Frontend.java:987] Analyzing query: create table
> alltypesinsert partitioned by (year, month) stored as parquet as /*
> +noclustered */
> select at1.id, at1.bool_col, at1.tinyint_col, at1.smallint_col, at1.int_col,
> at1.bigint_col,
> at1.float_col, at1.double_col, at1.date_string_col, at1.string_col,
> at1.timestamp_col,
> at1.year, at2.id as month
> from functional.alltypes at1, functional.alltypes at2
> I0510 13:46:03.025013 4238 jni-util.cc:230] java.lang.NullPointerException
> at
> org.apache.impala.analysis.SqlScanner.isReserved(SqlScanner.java:725)
> at
> org.apache.impala.analysis.SqlParser.getErrorMsg(SqlParser.java:1532)
> at org.apache.impala.service.Frontend.parse(Frontend.java:975)
> at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:990)
> at
> org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:156)
> I0510 13:46:03.124739 4238 status.cc:125] NullPointerException: null
> @ 0x18782ef impala::Status::Status()
> @ 0x1e55652 impala::JniUtil::GetJniExceptionMsg()
> @ 0x1d133ed impala::JniUtil::CallJniMethod<>()
> @ 0x1d10047 impala::Frontend::GetExecRequest()
> @ 0x1d3205a impala::ImpalaServer::ExecuteInternal()
> @ 0x1d31ba2 impala::ImpalaServer::Execute()
> @ 0x1d9be70 impala::ImpalaServer::query()
> @ 0x2ee378e beeswax::BeeswaxServiceProcessor::process_query()
> @ 0x2ee34dc beeswax::BeeswaxServiceProcessor::dispatchCall()
> @ 0x2ebcf9d impala::ImpalaServiceProcessor::dispatchCall()
> @ 0x1836690 apache::thrift::TDispatchProcessor::process()
> @ 0x1b9649d
> apache::thrift::server::TAcceptQueueServer::Task::run()
> @ 0x1b8d9c5 impala::ThriftThread::RunRunnable()
> @ 0x1b8f0c9 boost::_mfi::mf2<>::operator()()
> @ 0x1b8ef5f boost::_bi::list3<>::operator()<>()
> @ 0x1b8ecab boost::_bi::bind_t<>::operator()()
> @ 0x1b8ebbe
> boost::detail::function::void_function_obj_invoker0<>::invoke()
> @ 0x1bd3b1a boost::function0<>::operator()()
> @ 0x1ebec51 impala::Thread::SuperviseThread()
> @ 0x1ec6ded boost::_bi::list5<>::operator()<>()
> @ 0x1ec6d11 boost::_bi::bind_t<>::operator()()
> @ 0x1ec6cd4 boost::detail::thread_data<>::run()
> @ 0x31b3a4a thread_proxy
> @ 0x7fcf12d536ba start_thread
> @ 0x7fcf12a8941d clone
> I0510 13:46:03.124944 4238 impala-server.cc:1010] UnregisterQuery():
> query_id=6b4791bb7a54de54:16bdcba7
> I0510 13:46:03.124948 4238 impala-server.cc:1097] Cancel():
> query_id=6b4791bb7a54de54:16bdcba7
> {noformat}
> This is on commit hash 3e736450354e55244e16924cfeb223a30629351d . It looks
> like the code was added by IMPALA-3916
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail:
[jira] [Resolved] (IMPALA-7067) sleep(100000) command from test_shell_commandline.py can hang around and cause test_metrics_are_zero to fail
[
https://issues.apache.org/jira/browse/IMPALA-7067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-7067.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
Impala 2.13.0
> sleep(10) command from test_shell_commandline.py can hang around and
> cause test_metrics_are_zero to fail
>
>
> Key: IMPALA-7067
> URL: https://issues.apache.org/jira/browse/IMPALA-7067
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: flaky
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> {noformat}
> 03:25:47 [gw6] PASSED
> shell/test_shell_commandline.py::TestImpalaShell::test_cancellation
> ...
> 03:27:01 verifiers/test_verify_metrics.py:34: in test_metrics_are_zero
> 03:27:01 verifier.verify_metrics_are_zero()
> 03:27:01 verifiers/metric_verifier.py:47: in verify_metrics_are_zero
> 03:27:01 self.wait_for_metric(metric, 0, timeout)
> 03:27:01 verifiers/metric_verifier.py:62: in wait_for_metric
> 03:27:01 self.impalad_service.wait_for_metric_value(metric_name,
> expected_value, timeout)
> 03:27:01 common/impala_service.py:135: in wait_for_metric_value
> 03:27:01 json.dumps(self.read_debug_webpage('rpcz?json')))
> 03:27:01 E AssertionError: Metric value impala-server.mem-pool.total-bytes
> did not reach value 0 in 60s
> {noformat}
> I used the json dump from memz and the logs to trace it back to the
> sleep(10) query hanging around
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7073) Failed test: query_test.test_scanners.TestScannerReservation.test_scanners
[
https://issues.apache.org/jira/browse/IMPALA-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490275#comment-16490275
]
Tim Armstrong commented on IMPALA-7073:
---
I added this test so I'll take it for now. I'm guessing it's a out-of-date
profile, i.e. IMPALA-6338 but there's not enough context here to diagnose.
> Failed test: query_test.test_scanners.TestScannerReservation.test_scanners
> --
>
> Key: IMPALA-7073
> URL: https://issues.apache.org/jira/browse/IMPALA-7073
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0
>Reporter: Dimitris Tsirogiannis
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build, test-failure
>
> Possibly flaky test:
> {code:java}
> Stacktrace
> query_test/test_scanners.py:1064: in test_scanners
> self.run_test_case('QueryTest/scanner-reservation', vector)
> common/impala_test_suite.py:451: in run_test_case
> verify_runtime_profile(test_section['RUNTIME_PROFILE'],
> result.runtime_profile)
> common/test_result_verifier.py:590: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex:.*InitialRangeActualReservation.*Avg: 4.00 MB.*{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7023) TestInsertQueries.test_insert_overwrite fails by hitting memory limit
[
https://issues.apache.org/jira/browse/IMPALA-7023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16476086#comment-16476086
]
Tim Armstrong commented on IMPALA-7023:
---
So there's some bug or other issue causing those queries to be slow cleaning
themselves up?
Depending on whether we can/will fix that issue immediately, we could add a
wait_for_metric() call to the end of that test to make sure the query is
flushed out: http://gerrit.cloudera.org:8080/5493
> TestInsertQueries.test_insert_overwrite fails by hitting memory limit
> -
>
> Key: IMPALA-7023
> URL: https://issues.apache.org/jira/browse/IMPALA-7023
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Joe McDonnell
>Priority: Blocker
> Labels: broken-build
>
> This failure is seen on exhaustive builds on both master and 2.x:
> {noformat}
> Error Message
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: AnalysisException: Failed to
> evaluate expr: 20 CAUSED BY: InternalException: Memory limit exceeded: Error
> occurred on backend
> impala-boost-static-burst-slave-el7-03d4.vpc.cloudera.com:22000 by fragment
> 0:0 Memory left in process limit: -4.29 GB
> Query(d24ea53242b4cedc:fc8e0885): Total=0 Peak=0 : Total=0
> Peak=0Process: memory limit exceeded. Limit=12.00 GB Total=16.29 GB
> Peak=16.29 GB Buffer Pool: Free Buffers: Total=160.00 KB Buffer Pool:
> Clean Pages: Total=0 Buffer Pool: Unused Reservation: Total=-328.00 KB
> Data Stream Service Queue: Limit=614.40 MB Total=0 Peak=116.12 KB Data
> Stream Manager Early RPCs: Total=0 Peak=6.76 KB TCMalloc Overhead:
> Total=103.56 MB RequestPool=fe-eval-exprs: Total=0 Peak=52.83 KB
> Query(d24ea53242b4cedc:fc8e0885): Total=0 Peak=0
> RequestPool=default-pool: Total=5.20 GB Peak=5.20 GB
> Query(f4014f7bb49ea78:6b926b19): memory limit exceeded. Limit=64.00
> MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=1.76 GB Total=1.76 GB
> Peak=1.96 GB Query(2c44b65fbcb4e1ce:3d73badf): memory limit
> exceeded. Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB
> OtherMemory=1.04 GB Total=1.04 GB Peak=1.61 GB
> Query(214cc23c1376176f:7844977b): memory limit exceeded. Limit=64.00
> MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=1.23 GB Total=1.23 GB
> Peak=1.23 GB Query(8949bdf792a32ad2:33a36c03): memory limit
> exceeded. Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB
> OtherMemory=642.20 MB Total=642.20 MB Peak=642.20 MB
> Query(5412ff4e6065721:519d3e61): memory limit exceeded. Limit=64.00
> MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=556.49 MB Total=556.49
> MB Peak=556.49 MB Untracked Memory: Total=10.98 GB
> Stacktrace
> query_test/test_insert.py:132: in test_insert_overwrite
> multiple_impalad=vector.get_value('exec_option')['sync_ddl'] == 1)
> common/impala_test_suite.py:405: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:339: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:335: in execute_query_async
> return self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:460: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: AnalysisException: Failed to evaluate expr: 20
> E CAUSED BY: InternalException: Memory limit exceeded: Error occurred on
> backend impala-boost-static-burst-slave-el7-03d4.vpc.cloudera.com:22000 by
> fragment 0:0
> E Memory left in process limit: -4.29 GB
> E Query(d24ea53242b4cedc:fc8e0885): Total=0 Peak=0
> E : Total=0 Peak=0Process: memory limit exceeded. Limit=12.00 GB
> Total=16.29 GB Peak=16.29 GB
> E Buffer Pool: Free Buffers: Total=160.00 KB
> E Buffer Pool: Clean Pages: Total=0
> E Buffer Pool: Unused Reservation: Total=-328.00 KB
> E Data Stream Service Queue: Limit=614.40 MB Total=0 Peak=116.12 KB
> E Data Stream Manager Early RPCs: Total=0 Peak=6.76 KB
> E TCMalloc Overhead: Total=103.56 MB
> E RequestPool=fe-eval-exprs: Total=0 Peak=52.83 KB
> E
[jira] [Commented] (IMPALA-7049) Scan node reservation calculation seems off
[
https://issues.apache.org/jira/browse/IMPALA-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481320#comment-16481320
]
Tim Armstrong commented on IMPALA-7049:
---
Attached a profile from an affected query. It doesn't really make sense to me -
the plan looks like it's from before the below commit, but that's the commit
that added the error message:
{noformat}
commit fb5dc9eb484e54cf9f37d06168392c5bc2a0f4fe
Author: Tim Armstrong
Date: Sun Oct 29 12:38:47 2017 -0700
IMPALA-4835: switch I/O buffers to buffer pool
{noformat}
> Scan node reservation calculation seems off
> ---
>
> Key: IMPALA-7049
> URL: https://issues.apache.org/jira/browse/IMPALA-7049
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Michael Ho
>Assignee: Tim Armstrong
>Priority: Critical
> Attachments: profile.txt
>
>
> Running the query TPC-DS Q77a with a memory limit, we ran into the error
> *HDFS scan min reservation 0 must be >= min buffer size 8192*:
> {noformat}
> Query Type: QUERY
> Query State: EXCEPTION
> Query Status: HDFS scan min reservation 0 must be >= min buffer size 8192
> Sql Statement: /* Mem: 2375 MB. Coordinator: machine. */
> -- RESULT MISMATCH FROM ORIGINAL
> -- FIXED. TAKE ACTUAL RESULT AS EXPECTED
> with ss as
> (select s_store_sk,
> sum(ss_ext_sales_price) as sales,
> sum(ss_net_profit) as profit
> from store_sales,
> date_dim,
> store
> where ss_sold_date_sk = d_date_sk
>and cast(d_date as timestamp) between cast('2000-08-23' as timestamp)
> and (cast('2000-08-23' as timestamp) + interval 30 days)
>and ss_store_sk = s_store_sk
> group by s_store_sk)
> ,
> sr as
> (select s_store_sk,
> sum(sr_return_amt) as return_amt,
> sum(sr_net_loss) as profit_loss
> from store_returns,
> date_dim,
> store
> where sr_returned_date_sk = d_date_sk
>and cast(d_date as timestamp) between cast('2000-08-23' as timestamp)
> and (cast('2000-08-23' as timestamp) + interval 30 days)
>and sr_store_sk = s_store_sk
> group by s_store_sk),
> cs as
> (select cs_call_center_sk,
> sum(cs_ext_sales_price) as sales,
> sum(cs_net_profit) as profit
> from catalog_sales,
> date_dim
> where cs_sold_date_sk = d_date_sk
>and cast(d_date as timestamp) between cast('2000-08-23' as timestamp)
> and (cast('2000-08-23' as timestamp) + interval 30 days)
> group by cs_call_center_sk
> ),
> cr as
> (select cr_call_center_sk,
> sum(cr_return_amount) as return_amt,
> sum(cr_net_loss) as profit_loss
> from catalog_returns,
> date_dim
> where cr_returned_date_sk = d_date_sk
>and cast(d_date as timestamp) between cast('2000-08-23' as timestamp)
> and (cast('2000-08-23' as timestamp) + interval 30 days)
> group by cr_call_center_sk
> ),
> ws as
> ( select wp_web_page_sk,
> sum(ws_ext_sales_price) as sales,
> sum(ws_net_profit) as profit
> from web_sales,
> date_dim,
> web_page
> where ws_sold_date_sk = d_date_sk
>and cast(d_date as timestamp) between cast('2000-08-23' as timestamp)
> and (cast('2000-08-23' as timestamp) + interval 30 days)
>and ws_web_page_sk = wp_web_page_sk
> group by wp_web_page_sk),
> wr as
> (select wp_web_page_sk,
> sum(wr_return_amt) as return_amt,
> sum(wr_net_loss) as profit_loss
> from web_returns,
> date_dim,
> web_page
> where wr_returned_date_sk = d_date_sk
>and cast(d_date as timestamp) between cast('2000-08-23' as timestamp)
> and (cast('2000-08-23' as timestamp) + interval 30 days)
>and wr_web_page_sk = wp_web_page_sk
> group by wp_web_page_sk)
> ,
> results as
> (select channel
> , id
> , sum(sales) as sales
> , sum(return_amt) as return_amt
> , sum(profit) as profit
> from
> (select 'store channel' as channel
> , ss.s_store_sk as id
> , sales
> , coalesce(return_amt, 0) as return_amt
> , (profit - coalesce(profit_loss,0)) as profit
> from ss left join sr
> on ss.s_store_sk = sr.s_store_sk
> union all
> select 'catalog channel' as channel
> , cs_call_center_sk as id
> , sales
> , return_amt
> , (profit - profit_loss) as profit
> from cs
>, cr
> union all
> select 'web channel' as channel
> , ws.wp_web_page_sk as id
> , sales
> , coalesce(return_amt, 0) return_amt
> , (profit - coalesce(profit_loss,0)) as profit
> from ws left join wr
> on
[jira] [Resolved] (IMPALA-6995) False-positive DCHECK when converting whitespace-only strings to timestamp
[
https://issues.apache.org/jira/browse/IMPALA-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-6995.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
Impala 2.13.0
> False-positive DCHECK when converting whitespace-only strings to timestamp
> --
>
> Key: IMPALA-6995
> URL: https://issues.apache.org/jira/browse/IMPALA-6995
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: crash
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> {noformat}
> select cast(' ' as timestamp);
> {noformat}
> {noformat}
> F0508 14:32:07.245255 11824 timestamp-parse-util.cc:241] Check failed:
> dt_ctx->fmt_len > 0 (0 vs. 0)
> *** Check failure stack trace: ***
> @ 0x428956d google::LogMessage::Fail()
> @ 0x428ae12 google::LogMessage::SendToLog()
> @ 0x4288f47 google::LogMessage::Flush()
> @ 0x428c50e google::LogMessageFatal::~LogMessageFatal()
> @ 0x1c3f485 impala::TimestampParser::ParseFormatTokensByStr()
> @ 0x1c40553 impala::TimestampParser::Parse()
> @ 0x1c4712a impala::TimestampValue::Parse()
> @ 0x2e5d8fa impala::CastFunctions::CastToTimestampVal()
> @ 0x2e45322 impala::ScalarFnCall::InterpretEval<>()
> @ 0x2e27de5 impala::ScalarFnCall::GetTimestampVal()
> @ 0x2de72de impala::ScalarExprEvaluator::GetValue()
> @ 0x2de6e69 impala::ScalarExprEvaluator::GetValue()
> @ 0x1d1dbbf
> Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow
> @ 0x7fb7cc1d07e8 (unknown)
> Picked up JAVA_TOOL_OPTIONS:
> -agentlib:jdwp=transport=dt_socket,address=3,server=y,suspend=n
> Wrote minidump to
> /home/tarmstrong/Impala/incubator-impala/logs/cluster/minidumps/impalad/42afc7f9-5b4a-4ed7-b34ad782-d7904747.dmp
> {noformat}
> It seems to work fine on a release build.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-1034) Verify RSS stays within mem_limit + JVM heapsize
[ https://issues.apache.org/jira/browse/IMPALA-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong reassigned IMPALA-1034: - Assignee: (was: Tim Armstrong) > Verify RSS stays within mem_limit + JVM heapsize > > > Key: IMPALA-1034 > URL: https://issues.apache.org/jira/browse/IMPALA-1034 > Project: IMPALA > Issue Type: Test > Components: Backend >Affects Versions: Impala 1.3.1 >Reporter: Alan Choi >Priority: Major > Labels: resource-management > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-6969) Profile doesn't include the reason that a query couldn't be dequeued from admission controller
[
https://issues.apache.org/jira/browse/IMPALA-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-6969 started by Tim Armstrong.
-
> Profile doesn't include the reason that a query couldn't be dequeued from
> admission controller
> --
>
> Key: IMPALA-6969
> URL: https://issues.apache.org/jira/browse/IMPALA-6969
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Labels: admission-control, observability
>
> I noticed this while playing around on a local minicluster with AC enabled.
> The admission controller adds the reason for initial queuing to the profile,
> but does not expose why the query couldn't execute when it got to the head of
> the line. E.g. if it was initially queued because the queue was non-empty but
> then couldn't execute once it got to the head of the line because of memory.
> {noformat}
> Request Pool: root.queueA
> Admission result: Admitted (queued)
> Admission queue details: waited 1130 ms, reason: queue is not empty (size
> 4); queued queries are executed first
> {noformat}
> We should still include the initial reason for queuing, but also include the
> most reason for queuing once it got to the head of the line. It's probably
> most useful to keep the profile updated with the latest reason at all times
> (since the details can change while the query is at the head of the line).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6969) Profile doesn't include the reason that a query couldn't be dequeued from admission controller
[
https://issues.apache.org/jira/browse/IMPALA-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16476583#comment-16476583
]
Tim Armstrong commented on IMPALA-6969:
---
I prototyped a fix of this. It touches the same code as the async queuing, so I
won't finish it off until that is merged.
> Profile doesn't include the reason that a query couldn't be dequeued from
> admission controller
> --
>
> Key: IMPALA-6969
> URL: https://issues.apache.org/jira/browse/IMPALA-6969
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Labels: admission-control, observability
>
> I noticed this while playing around on a local minicluster with AC enabled.
> The admission controller adds the reason for initial queuing to the profile,
> but does not expose why the query couldn't execute when it got to the head of
> the line. E.g. if it was initially queued because the queue was non-empty but
> then couldn't execute once it got to the head of the line because of memory.
> {noformat}
> Request Pool: root.queueA
> Admission result: Admitted (queued)
> Admission queue details: waited 1130 ms, reason: queue is not empty (size
> 4); queued queries are executed first
> {noformat}
> We should still include the initial reason for queuing, but also include the
> most reason for queuing once it got to the head of the line. It's probably
> most useful to keep the profile updated with the latest reason at all times
> (since the details can change while the query is at the head of the line).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work stopped] (IMPALA-6969) Profile doesn't include the reason that a query couldn't be dequeued from admission controller
[
https://issues.apache.org/jira/browse/IMPALA-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-6969 stopped by Tim Armstrong.
-
> Profile doesn't include the reason that a query couldn't be dequeued from
> admission controller
> --
>
> Key: IMPALA-6969
> URL: https://issues.apache.org/jira/browse/IMPALA-6969
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Labels: admission-control, observability
>
> I noticed this while playing around on a local minicluster with AC enabled.
> The admission controller adds the reason for initial queuing to the profile,
> but does not expose why the query couldn't execute when it got to the head of
> the line. E.g. if it was initially queued because the queue was non-empty but
> then couldn't execute once it got to the head of the line because of memory.
> {noformat}
> Request Pool: root.queueA
> Admission result: Admitted (queued)
> Admission queue details: waited 1130 ms, reason: queue is not empty (size
> 4); queued queries are executed first
> {noformat}
> We should still include the initial reason for queuing, but also include the
> most reason for queuing once it got to the head of the line. It's probably
> most useful to keep the profile updated with the latest reason at all times
> (since the details can change while the query is at the head of the line).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7058) Crash in exhaustive tests for rhel7
[
https://issues.apache.org/jira/browse/IMPALA-7058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487648#comment-16487648
]
Tim Armstrong commented on IMPALA-7058:
---
I'll push out a fix to disable the fuzz test. We should keep the product
improvements in IMPALA-3833 - it looks like there were a couple of rare cases
we didn't fix the first time around.
> Crash in exhaustive tests for rhel7
>
>
> Key: IMPALA-7058
> URL: https://issues.apache.org/jira/browse/IMPALA-7058
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0
>Reporter: Dimitris Tsirogiannis
>Assignee: Pranay Singh
>Priority: Blocker
> Labels: broken-build, crash
>
> The backtrace is here:
> {code:java}
> #7 0x02d89a84 in
> impala::DelimitedTextParser::ParseFieldLocations (this=0xcf539a0,
> max_tuples=1, remaining_len=-102, byte_buffer_ptr=0x7fc6b764dad0,
> row_end_locations=0x7fc6b764dac0, field_locations=0x10034000,
> num_tuples=0x7fc6b764dacc, num_fields=0x7fc6b764dac8,
> next_column_start=0x7fc6b764dad8) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/delimited-text-parser.cc:205
> #8 0x01fdb641 in impala::HdfsSequenceScanner::ProcessRange
> (this=0x15515f80, row_batch=0xcf54800) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-sequence-scanner.cc:352
> #9 0x02d7a20e in impala::BaseSequenceScanner::GetNextInternal
> (this=0x15515f80, row_batch=0xcf54800) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/base-sequence-scanner.cc:181
> #10 0x01fb1ff0 in impala::HdfsScanner::ProcessSplit (this=0x15515f80)
> at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scanner.cc:134
> #11 0x01f89258 in impala::HdfsScanNode::ProcessSplit
> (this=0x2a4a8800, filter_ctxs=..., expr_results_pool=0x7fc6b764e4b0,
> scan_range=0x13f5f8700, scanner_thread_reservation=0x7fc6b764e428) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scan-node.cc:453
> #12 0x01f885f9 in impala::HdfsScanNode::ScannerThread
> (this=0x2a4a8800, first_thread=false, scanner_thread_reservation=32768) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scan-node.cc:360
> #13 0x01f87a6c in impala::HdfsScanNode::::operator()(void)
> const (__closure=0x7fc6b764ebe8) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scan-node.cc:292
> #14 0x01f89ac8 in
> boost::detail::function::void_function_obj_invoker0,
> void>::invoke(boost::detail::function::function_buffer &)
> (function_obj_ptr=...) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #15 0x01bf0b28 in boost::function0::operator()
> (this=0x7fc6b764ebe0) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #16 0x01edc57f in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*) (name=..., category=..., functor=...,
> parent_thread_info=0x7fc6b9e53890, thread_started=0x7fc6b9e527c0) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/util/thread.cc:356
> #17 0x01ee471b in boost::_bi::list5 boost::_bi::value, boost::_bi::value,
> boost::_bi::value,
> boost::_bi::value >::operator() const&, std::string const&, boost::function, impala::ThreadDebugInfo
> const*, impala::Promise*), boost::_bi::list0>(boost::_bi::type,
> void (*&)(std::string const&, std::string const&, boost::function,
> impala::ThreadDebugInfo const*, impala::Promise*), boost::_bi::list0&,
> int) (this=0x2a370fc0, f=@0x2a370fb8: 0x1edc218
> boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*)>, a=...) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525
> #18 0x01ee463f in boost::_bi::bind_t const&, std::string const&, boost::function, impala::ThreadDebugInfo
> const*, impala::Promise*),
> boost::_bi::list5 boost::_bi::value, boost::_bi::value,
>
[jira] [Updated] (IMPALA-7058) RC and Seq fuzz tests cause crash
[
https://issues.apache.org/jira/browse/IMPALA-7058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7058:
--
Summary: RC and Seq fuzz tests cause crash (was: Crash in exhaustive tests
for rhel7 )
> RC and Seq fuzz tests cause crash
> -
>
> Key: IMPALA-7058
> URL: https://issues.apache.org/jira/browse/IMPALA-7058
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0
>Reporter: Dimitris Tsirogiannis
>Assignee: Pranay Singh
>Priority: Blocker
> Labels: broken-build, crash
>
> The backtrace is here:
> {code:java}
> #7 0x02d89a84 in
> impala::DelimitedTextParser::ParseFieldLocations (this=0xcf539a0,
> max_tuples=1, remaining_len=-102, byte_buffer_ptr=0x7fc6b764dad0,
> row_end_locations=0x7fc6b764dac0, field_locations=0x10034000,
> num_tuples=0x7fc6b764dacc, num_fields=0x7fc6b764dac8,
> next_column_start=0x7fc6b764dad8) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/delimited-text-parser.cc:205
> #8 0x01fdb641 in impala::HdfsSequenceScanner::ProcessRange
> (this=0x15515f80, row_batch=0xcf54800) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-sequence-scanner.cc:352
> #9 0x02d7a20e in impala::BaseSequenceScanner::GetNextInternal
> (this=0x15515f80, row_batch=0xcf54800) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/base-sequence-scanner.cc:181
> #10 0x01fb1ff0 in impala::HdfsScanner::ProcessSplit (this=0x15515f80)
> at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scanner.cc:134
> #11 0x01f89258 in impala::HdfsScanNode::ProcessSplit
> (this=0x2a4a8800, filter_ctxs=..., expr_results_pool=0x7fc6b764e4b0,
> scan_range=0x13f5f8700, scanner_thread_reservation=0x7fc6b764e428) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scan-node.cc:453
> #12 0x01f885f9 in impala::HdfsScanNode::ScannerThread
> (this=0x2a4a8800, first_thread=false, scanner_thread_reservation=32768) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scan-node.cc:360
> #13 0x01f87a6c in impala::HdfsScanNode::::operator()(void)
> const (__closure=0x7fc6b764ebe8) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scan-node.cc:292
> #14 0x01f89ac8 in
> boost::detail::function::void_function_obj_invoker0,
> void>::invoke(boost::detail::function::function_buffer &)
> (function_obj_ptr=...) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #15 0x01bf0b28 in boost::function0::operator()
> (this=0x7fc6b764ebe0) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #16 0x01edc57f in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*) (name=..., category=..., functor=...,
> parent_thread_info=0x7fc6b9e53890, thread_started=0x7fc6b9e527c0) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/util/thread.cc:356
> #17 0x01ee471b in boost::_bi::list5 boost::_bi::value, boost::_bi::value,
> boost::_bi::value,
> boost::_bi::value >::operator() const&, std::string const&, boost::function, impala::ThreadDebugInfo
> const*, impala::Promise*), boost::_bi::list0>(boost::_bi::type,
> void (*&)(std::string const&, std::string const&, boost::function,
> impala::ThreadDebugInfo const*, impala::Promise*), boost::_bi::list0&,
> int) (this=0x2a370fc0, f=@0x2a370fb8: 0x1edc218
> boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*)>, a=...) at
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525
> #18 0x01ee463f in boost::_bi::bind_t const&, std::string const&, boost::function, impala::ThreadDebugInfo
> const*, impala::Promise*),
> boost::_bi::list5 boost::_bi::value, boost::_bi::value,
> boost::_bi::value,
> boost::_bi::value > >::operator()() (this=0x2a370fb8)
> at
>
[jira] [Resolved] (IMPALA-6960) TestMtDopParquet.test_parquet_filtering flaky
[
https://issues.apache.org/jira/browse/IMPALA-6960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-6960.
---
Resolution: Duplicate
I'll mark this as a dupe. If we see this specific failure happen again, I'll
look at mitigating it.
> TestMtDopParquet.test_parquet_filtering flaky
> -
>
> Key: IMPALA-6960
> URL: https://issues.apache.org/jira/browse/IMPALA-6960
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Labels: flaky
> Attachments: TEST-impala-parallel.log, TEST-impala-parallel.xml,
> impalad.ip-172-31-21-122.ubuntu.log.INFO.20180502-174947.82370.gz,
> impalad.ip-172-31-21-122.ubuntu.log.INFO.20180502-174948.82449,
> impalad.ip-172-31-21-122.ubuntu.log.INFO.20180502-174949.82508
>
>
> {noformat}
> 19:28:35 TestMtDopParquet.test_parquet_filtering[mt_dop: 2 | exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> 19:28:35 [gw0] linux2 -- Python 2.7.12
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 19:28:35 query_test/test_mt_dop.py:103: in test_parquet_filtering
> 19:28:35 self.run_test_case('QueryTest/parquet-filtering', vector)
> 19:28:35 common/impala_test_suite.py:451: in run_test_case
> 19:28:35 verify_runtime_profile(test_section['RUNTIME_PROFILE'],
> result.runtime_profile)
> 19:28:35 common/test_result_verifier.py:599: in verify_runtime_profile
> 19:28:35 "\n\nPROFILE:\n%s\n" % (function, field, expected_value,
> actual_value, actual))
> 19:28:35 E AssertionError: Aggregation of SUM over NumRowGroups did not
> match expected results.
> 19:28:35 E EXPECTED VALUE:
> 19:28:35 E 24
> 19:28:35 E
> 19:28:35 E ACTUAL VALUE:
> 19:28:35 E 21
> ...
> 19:28:35 -- executing against localhost:21000
> 19:28:35 select count(*) from functional_parquet.alltypes where
> date_string_col like '%/10';
> 19:28:35
> 19:28:35 = 1 failed, 1899 passed, 60 skipped, 44 xfailed, 1 xpassed in
> 2187.10 seconds ==
> {noformat}
> Query ID is be45f2c7cbf5354f:44e7abd3
> Could be the same problem as IMPALA-6338 with a missing profile. I'm going to
> dig a little more to confirm that one of the fragment instance profiles i
> missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7008) TestSpillingDebugActionDimensions.test_spilling test setup fails
[
https://issues.apache.org/jira/browse/IMPALA-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-7008:
-
Assignee: David Knupp
> TestSpillingDebugActionDimensions.test_spilling test setup fails
>
>
> Key: IMPALA-7008
> URL: https://issues.apache.org/jira/browse/IMPALA-7008
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.0, Impala 2.13.0
>Reporter: Sailesh Mukil
>Assignee: David Knupp
>Priority: Blocker
> Labels: broken-build, flaky
>
> We've seen multiple instances of this test failing with the following error:
> {code:java}
> Error Message
> test setup failure
> Stacktrace
> Slave 'gw0' crashed while running
> "query_test/test_spilling.py::TestSpillingDebugActionDimensions::()::test_spilling[exec_option:
> {'debug_action': None, 'default_spillable_buffer_size': '256k'} |
> table_format: parquet/none]"
> {code}
> We need to investigate why this is happening.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5212) consider switching to pread by default
[ https://issues.apache.org/jira/browse/IMPALA-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487761#comment-16487761 ] Tim Armstrong commented on IMPALA-5212: --- I think it's possible that this could affect remote read performance for larger scan ranges > 8mb - I think the non-pread implementation has some kind of prefetching built-in. > consider switching to pread by default > -- > > Key: IMPALA-5212 > URL: https://issues.apache.org/jira/browse/IMPALA-5212 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Silvius Rus >Assignee: Joe McDonnell >Priority: Major > > 1) Review the current HDFS tests. Validate that we have sufficient coverage, > add coverage if need be. > 2) Switch to pread as default. > 3) Consider deprecating the old read path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-7028) TestKuduOperations.test_kudu_update fails due to wrong NumModifiedRows
[
https://issues.apache.org/jira/browse/IMPALA-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487685#comment-16487685
]
Tim Armstrong edited comment on IMPALA-7028 at 5/23/18 6:44 PM:
-Probably the profile bug-
was (Author: tarmstrong):
Probably the profile bug
> TestKuduOperations.test_kudu_update fails due to wrong NumModifiedRows
> --
>
> Key: IMPALA-7028
> URL: https://issues.apache.org/jira/browse/IMPALA-7028
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
>
> Seen once on master exhaustive:
> {noformat}
> query_test/test_kudu.py:92: in test_kudu_update
> self.run_test_case('QueryTest/kudu_update', vector,
> use_db=unique_database)
> common/impala_test_suite.py:451: in run_test_case
> verify_runtime_profile(test_section['RUNTIME_PROFILE'],
> result.runtime_profile)
> common/test_result_verifier.py:590: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E NumModifiedRows: 7300
> ...
> # From the actual profile:
> E Partition: Default
> E NumModifiedRows: 32
> E NumRowErrors: 0{noformat}
> Will add a comment with the profile.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-6338) test_profile_fragment_instances failing
[
https://issues.apache.org/jira/browse/IMPALA-6338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-6338:
--
Attachment: kudu-update-profile.txt
> test_profile_fragment_instances failing
> ---
>
> Key: IMPALA-6338
> URL: https://issues.apache.org/jira/browse/IMPALA-6338
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0, Impala 2.12.0
>Reporter: David Knupp
>Assignee: Thomas Tauber-Marshall
>Priority: Critical
> Labels: broken-build, flaky
> Attachments: kudu-update-profile.txt, profile-failure.txt,
> profile-success.txt
>
>
> Stack trace:
> {noformat}
> query_test/test_observability.py:123: in test_profile_fragment_instances
> assert results.runtime_profile.count("HDFS_SCAN_NODE") == 12
> E assert 11 == 12
> E+ where 11 = 0x68bd2f0>('HDFS_SCAN_NODE')
> E+where = 'Query
> (id=ae4cee91aafc5c6c:11b545c6):\n DEBUG MODE WARNING: Query profile
> created while running a DEBUG buil...ontextSwitches: 0 (0)\n -
> TotalRawHdfsReadTime(*): 5s784ms\n - TotalReadThroughput: 17.33
> MB/sec\n'.count
> E+ where 'Query (id=ae4cee91aafc5c6c:11b545c6):\n DEBUG
> MODE WARNING: Query profile created while running a DEBUG
> buil...ontextSwitches: 0 (0)\n - TotalRawHdfsReadTime(*): 5s784ms\n
> - TotalReadThroughput: 17.33 MB/sec\n' =
> 0x6322e10>.runtime_profile
> {noformat}
> Query:
> {noformat}
> with l as (select * from tpch.lineitem UNION ALL select * from tpch.lineitem)
> select STRAIGHT_JOIN count(*) from (select * from tpch.lineitem a
> LIMIT 1) a
> join (select * from l LIMIT 200) b on a.l_orderkey =
> -b.l_orderkey;
> {noformat}
> Summary:
> {noformat}
> Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem
> Est. Peak Mem Detail
>
> 05:AGGREGATE 1 0.000ns 0.000ns 1 1 28.00 KB
> 10.00 MB FINALIZE
> 04:HASH JOIN 1 15.000ms 15.000ms 0 1 141.06 MB
> 17.00 MB INNER JOIN, BROADCAST
> |--08:EXCHANGE1 4s153ms 4s153ms 2.00M 2.00M 0
> 0 UNPARTITIONED
> | 07:EXCHANGE1 3s783ms 3s783ms 2.00M 2.00M 0
> 0 UNPARTITIONED
> | 01:UNION 3 17.000ms 28.001ms 3.03M 2.00M 0
> 0
> | |--03:SCAN HDFS3 0.000ns 0.000ns 0 6.00M 0
> 176.00 MB tpch.lineitem
> | 02:SCAN HDFS 3 6s133ms 6s948ms 3.03M 6.00M 24.02 MB
> 176.00 MB tpch.lineitem
> 06:EXCHANGE 1 5s655ms 5s655ms 1 1 0
> 0 UNPARTITIONED
> 00:SCAN HDFS 3 4s077ms 6s207ms 2 1 16.05 MB
> 176.00 MB tpch.lineitem a
> {noformat}
> Plan:
> {noformat}
>
> Max Per-Host Resource Reservation: Memory=17.00MB
> Per-Host Resource Estimates: Memory=379.00MB
> F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=27.00MB mem-reservation=17.00MB
> PLAN-ROOT SINK
> | mem-estimate=0B mem-reservation=0B
> |
> 05:AGGREGATE [FINALIZE]
> | output: count(*)
> | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
> | tuple-ids=7 row-size=8B cardinality=1
> |
> 04:HASH JOIN [INNER JOIN, BROADCAST]
> | hash predicates: a.l_orderkey = -1 * l_orderkey
> | fk/pk conjuncts: assumed fk/pk
> | runtime filters: RF000[bloom] <- -1 * l_orderkey
> | mem-estimate=17.00MB mem-reservation=17.00MB spill-buffer=1.00MB
> | tuple-ids=0,4 row-size=16B cardinality=1
> |
> |--08:EXCHANGE [UNPARTITIONED]
> | | mem-estimate=0B mem-reservation=0B
> | | tuple-ids=4 row-size=8B cardinality=200
> | |
> | F05:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=0B mem-reservation=0B
> | 07:EXCHANGE [UNPARTITIONED]
> | | limit: 200
> | | mem-estimate=0B mem-reservation=0B
> | | tuple-ids=4 row-size=8B cardinality=200
> | |
> | F04:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
> | Per-Host Resources: mem-estimate=176.00MB mem-reservation=0B
> | 01:UNION
> | | pass-through-operands: all
> | | limit: 200
> | | mem-estimate=0B mem-reservation=0B
> | | tuple-ids=4 row-size=8B cardinality=200
> | |
> | |--03:SCAN HDFS [tpch.lineitem, RANDOM]
> | | partitions=1/1 files=1 size=718.94MB
> | | stored statistics:
> | | table: rows=6001215 size=718.94MB
> |
[jira] [Commented] (IMPALA-7028) TestKuduOperations.test_kudu_update fails due to wrong NumModifiedRows
[
https://issues.apache.org/jira/browse/IMPALA-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487846#comment-16487846
]
Tim Armstrong commented on IMPALA-7028:
---
According to the profile, all of the Kudu scan nodes returned only a fraction
of the expected rows. I don't think the profile bug applies since it's a
successful DML, which waits for all the final status reports to be sent in.
{noformat}
18:26:14 E KUDU_SCAN_NODE (id=0):(Total: 4.999ms, non-child: 4.999ms,
% non-child: 100.00%)
18:26:14 E - BytesRead: 0
18:26:14 E - CollectionItemsRead: 0 (0)
18:26:14 E - KuduRemoteScanTokens: 0 (0)
18:26:14 E - NumScannerThreadsStarted: 1 (1)
18:26:14 E - PeakMemoryUsage: 12.00 KB (12288)
18:26:14 E - RowsRead: 10 (10)
18:26:14 E - RowsReturned: 10 (10)
18:26:14 E - RowsReturnedRate: 2.00 K/sec
18:26:14 E - ScanRangesComplete: 1 (1)
18:26:14 E - ScannerThreadsInvoluntaryContextSwitches: 4 (4)
18:26:14 E - ScannerThreadsTotalWallClockTime: 3.999ms
18:26:14 E- MaterializeTupleTime(*): 1.999ms
18:26:14 E- ScannerThreadsSysTime: 0.000ns
18:26:14 E- ScannerThreadsUserTime: 999.000us
18:26:14 E - ScannerThreadsVoluntaryContextSwitches: 3 (3)
18:26:14 E - TotalKuduScanRoundTrips: 1 (1)
18:26:14 E - TotalReadThroughput: 0.00 /sec
{noformat}
[~twmarshall] this looks like another case of Kudu scans mysteriously not
returning the expected number of rows. Are we tracking those with a single
JIRA. I guess the rows were inserted by the previous statement, run against the
same coordinator.
{noformat}
18:26:14 -- executing against localhost:21000
18:26:14 insert into tdata
18:26:14 select id, string_col, float_col, bigint_col, string_col, bool_col,
tinyint_col,
18:26:14 smallint_col, double_col, NULL, NULL, NULL from
functional_kudu.alltypes;
18:26:14
18:26:14 -- executing against localhost:21000
18:26:14 update tdata set vali = -1;
{noformat}
> TestKuduOperations.test_kudu_update fails due to wrong NumModifiedRows
> --
>
> Key: IMPALA-7028
> URL: https://issues.apache.org/jira/browse/IMPALA-7028
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
>
> Seen once on master exhaustive:
> {noformat}
> query_test/test_kudu.py:92: in test_kudu_update
> self.run_test_case('QueryTest/kudu_update', vector,
> use_db=unique_database)
> common/impala_test_suite.py:451: in run_test_case
> verify_runtime_profile(test_section['RUNTIME_PROFILE'],
> result.runtime_profile)
> common/test_result_verifier.py:590: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E NumModifiedRows: 7300
> ...
> # From the actual profile:
> E Partition: Default
> E NumModifiedRows: 32
> E NumRowErrors: 0{noformat}
> Will add a comment with the profile.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7028) TestKuduOperations.test_kudu_update fails due to wrong NumModifiedRows
[
https://issues.apache.org/jira/browse/IMPALA-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7028:
--
Labels: broken-build correctness flaky (was: broken-build flaky)
> TestKuduOperations.test_kudu_update fails due to wrong NumModifiedRows
> --
>
> Key: IMPALA-7028
> URL: https://issues.apache.org/jira/browse/IMPALA-7028
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, correctness, flaky
> Attachments: kudu-update-profile.txt
>
>
> Seen once on master exhaustive:
> {noformat}
> query_test/test_kudu.py:92: in test_kudu_update
> self.run_test_case('QueryTest/kudu_update', vector,
> use_db=unique_database)
> common/impala_test_suite.py:451: in run_test_case
> verify_runtime_profile(test_section['RUNTIME_PROFILE'],
> result.runtime_profile)
> common/test_result_verifier.py:590: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E NumModifiedRows: 7300
> ...
> # From the actual profile:
> E Partition: Default
> E NumModifiedRows: 32
> E NumRowErrors: 0{noformat}
> Will add a comment with the profile.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7028) TestKuduOperations.test_kudu_update fails due to wrong NumModifiedRows
[
https://issues.apache.org/jira/browse/IMPALA-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7028:
--
Attachment: kudu-update-profile.txt
> TestKuduOperations.test_kudu_update fails due to wrong NumModifiedRows
> --
>
> Key: IMPALA-7028
> URL: https://issues.apache.org/jira/browse/IMPALA-7028
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, correctness, flaky
> Attachments: kudu-update-profile.txt
>
>
> Seen once on master exhaustive:
> {noformat}
> query_test/test_kudu.py:92: in test_kudu_update
> self.run_test_case('QueryTest/kudu_update', vector,
> use_db=unique_database)
> common/impala_test_suite.py:451: in run_test_case
> verify_runtime_profile(test_section['RUNTIME_PROFILE'],
> result.runtime_profile)
> common/test_result_verifier.py:590: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E NumModifiedRows: 7300
> ...
> # From the actual profile:
> E Partition: Default
> E NumModifiedRows: 32
> E NumRowErrors: 0{noformat}
> Will add a comment with the profile.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7156) Many authorization tests failed with "User 'jenkins' does not have privileges to access"
[
https://issues.apache.org/jira/browse/IMPALA-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7156:
--
Attachment: 0001-Fix-IMPALA-7156-breakage-in-Sentry.patch
> Many authorization tests failed with "User 'jenkins' does not have privileges
> to access"
>
>
> Key: IMPALA-7156
> URL: https://issues.apache.org/jira/browse/IMPALA-7156
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Tim Armstrong
>Assignee: Fredy Wijaya
>Priority: Blocker
> Labels: broken-build, flaky
> Attachments: 0001-Fix-IMPALA-7156-breakage-in-Sentry.patch,
> jenkins-priv-logs.tar.gz
>
>
> Failed tests:
> {noformat}
>
> authorization.test_grant_revoke.TestGrantRevoke.test_role_privilege_case[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelInsert[1]
> org.apache.impala.analysis.AuthorizationTest.TestFunction[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropDatabase[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelSelect[1]
> org.apache.impala.analysis.AuthorizationTest.TestTruncate[1]
> org.apache.impala.analysis.AuthorizationTest.TestShowCreateTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestShortUsernameUsed[1]
> org.apache.impala.analysis.AuthorizationTest.TestComputeStatsTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestHs2GetColumns[1]
> org.apache.impala.analysis.AuthorizationTest.TestLoad[1]
> org.apache.impala.analysis.AuthorizationTest.TestCreateDatabase[1]
> org.apache.impala.analysis.AuthorizationTest.TestInsert[1]
> org.apache.impala.analysis.AuthorizationTest.TestSelect[1]
> org.apache.impala.analysis.AuthorizationTest.TestCreateView[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelAlter[1]
> org.apache.impala.analysis.AuthorizationTest.TestResetMetadata[1]
> org.apache.impala.analysis.AuthorizationTest.TestExplain[1]
> org.apache.impala.analysis.AuthorizationTest.TestCommentOn[1]
> org.apache.impala.analysis.AuthorizationTest.TestWithClause[1]
> org.apache.impala.analysis.AuthorizationTest.TestAlterView[1]
> org.apache.impala.analysis.AuthorizationTest.TestShowPermissions[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelDrop[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropView[1]
> org.apache.impala.analysis.AuthorizationTest.TestShowDbResultsFiltered[1]
>
> org.apache.impala.analysis.AuthorizationTest.TestShowTableResultsFiltered[1]
> org.apache.impala.analysis.AuthorizationTest.TestUnion[1]
> org.apache.impala.analysis.AuthorizationTest.TestUseDb[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropStats[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestHs2GetSchema[1]
> org.apache.impala.analysis.AuthorizationTest.TestHs2GetTables[1]
> org.apache.impala.analysis.AuthorizationTest.TestDescribeDb[1]
> org.apache.impala.analysis.AuthorizationTest.TestAlterTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestDescribe[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelCreate[1]
> org.apache.impala.analysis.AuthorizationTest.TestCreateTable[1]
> {noformat}
> The errors are all along the lines of:
> {noformat}
> org.apache.impala.catalog.AuthorizationException: User 'jenkins' does not
> have privileges to execute 'INSERT' on: functional_parquet.alltypes
> at
> org.apache.impala.authorization.AuthorizationChecker.checkAccess(AuthorizationChecker.java:146)
> at
> org.apache.impala.analysis.AnalysisContext.authorizePrivilegeRequest(AnalysisContext.java:567)
> at
> org.apache.impala.analysis.AnalysisContext.authorize(AnalysisContext.java:530)
> at
> org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:409)
> at
> org.apache.impala.common.FrontendTestBase.parseAndAnalyze(FrontendTestBase.java:429)
> at
> org.apache.impala.analysis.AuthorizationTest.AuthzOk(AuthorizationTest.java:2841)
> at
> org.apache.impala.analysis.AuthorizationTest.AuthzOk(AuthorizationTest.java:2836)
> at
> org.apache.impala.analysis.AuthorizationTest.AuthzOk(AuthorizationTest.java:2832)
> at
> org.apache.impala.analysis.AuthorizationTest.TestTruncate(AuthorizationTest.java:1417)
> {noformat}
> {noformat}
> java.lang.AssertionError:
[jira] [Commented] (IMPALA-7156) Many authorization tests failed with "User 'jenkins' does not have privileges to access"
[
https://issues.apache.org/jira/browse/IMPALA-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507215#comment-16507215
]
Tim Armstrong commented on IMPALA-7156:
---
The attached patch is sufficient to fix the tests.
> Many authorization tests failed with "User 'jenkins' does not have privileges
> to access"
>
>
> Key: IMPALA-7156
> URL: https://issues.apache.org/jira/browse/IMPALA-7156
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Tim Armstrong
>Assignee: Fredy Wijaya
>Priority: Blocker
> Labels: broken-build, flaky
> Attachments: 0001-Fix-IMPALA-7156-breakage-in-Sentry.patch,
> jenkins-priv-logs.tar.gz
>
>
> Failed tests:
> {noformat}
>
> authorization.test_grant_revoke.TestGrantRevoke.test_role_privilege_case[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelInsert[1]
> org.apache.impala.analysis.AuthorizationTest.TestFunction[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropDatabase[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelSelect[1]
> org.apache.impala.analysis.AuthorizationTest.TestTruncate[1]
> org.apache.impala.analysis.AuthorizationTest.TestShowCreateTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestShortUsernameUsed[1]
> org.apache.impala.analysis.AuthorizationTest.TestComputeStatsTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestHs2GetColumns[1]
> org.apache.impala.analysis.AuthorizationTest.TestLoad[1]
> org.apache.impala.analysis.AuthorizationTest.TestCreateDatabase[1]
> org.apache.impala.analysis.AuthorizationTest.TestInsert[1]
> org.apache.impala.analysis.AuthorizationTest.TestSelect[1]
> org.apache.impala.analysis.AuthorizationTest.TestCreateView[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelAlter[1]
> org.apache.impala.analysis.AuthorizationTest.TestResetMetadata[1]
> org.apache.impala.analysis.AuthorizationTest.TestExplain[1]
> org.apache.impala.analysis.AuthorizationTest.TestCommentOn[1]
> org.apache.impala.analysis.AuthorizationTest.TestWithClause[1]
> org.apache.impala.analysis.AuthorizationTest.TestAlterView[1]
> org.apache.impala.analysis.AuthorizationTest.TestShowPermissions[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelDrop[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropView[1]
> org.apache.impala.analysis.AuthorizationTest.TestShowDbResultsFiltered[1]
>
> org.apache.impala.analysis.AuthorizationTest.TestShowTableResultsFiltered[1]
> org.apache.impala.analysis.AuthorizationTest.TestUnion[1]
> org.apache.impala.analysis.AuthorizationTest.TestUseDb[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropStats[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestHs2GetSchema[1]
> org.apache.impala.analysis.AuthorizationTest.TestHs2GetTables[1]
> org.apache.impala.analysis.AuthorizationTest.TestDescribeDb[1]
> org.apache.impala.analysis.AuthorizationTest.TestAlterTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestDescribe[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelCreate[1]
> org.apache.impala.analysis.AuthorizationTest.TestCreateTable[1]
> {noformat}
> The errors are all along the lines of:
> {noformat}
> org.apache.impala.catalog.AuthorizationException: User 'jenkins' does not
> have privileges to execute 'INSERT' on: functional_parquet.alltypes
> at
> org.apache.impala.authorization.AuthorizationChecker.checkAccess(AuthorizationChecker.java:146)
> at
> org.apache.impala.analysis.AnalysisContext.authorizePrivilegeRequest(AnalysisContext.java:567)
> at
> org.apache.impala.analysis.AnalysisContext.authorize(AnalysisContext.java:530)
> at
> org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:409)
> at
> org.apache.impala.common.FrontendTestBase.parseAndAnalyze(FrontendTestBase.java:429)
> at
> org.apache.impala.analysis.AuthorizationTest.AuthzOk(AuthorizationTest.java:2841)
> at
> org.apache.impala.analysis.AuthorizationTest.AuthzOk(AuthorizationTest.java:2836)
> at
> org.apache.impala.analysis.AuthorizationTest.AuthzOk(AuthorizationTest.java:2832)
> at
> org.apache.impala.analysis.AuthorizationTest.TestTruncate(AuthorizationTest.java:1417)
> {noformat}
> {noformat}
>
[jira] [Commented] (IMPALA-7156) Many authorization tests failed with "User 'jenkins' does not have privileges to access"
[
https://issues.apache.org/jira/browse/IMPALA-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507056#comment-16507056
]
Tim Armstrong commented on IMPALA-7156:
---
I'm testing to see if reverting the Sentry dependency to an older version can
unblock us temporarily, would love to have passing builds over the weekend.
> Many authorization tests failed with "User 'jenkins' does not have privileges
> to access"
>
>
> Key: IMPALA-7156
> URL: https://issues.apache.org/jira/browse/IMPALA-7156
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Tim Armstrong
>Assignee: Fredy Wijaya
>Priority: Blocker
> Labels: broken-build, flaky
> Attachments: jenkins-priv-logs.tar.gz
>
>
> Failed tests:
> {noformat}
>
> authorization.test_grant_revoke.TestGrantRevoke.test_role_privilege_case[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelInsert[1]
> org.apache.impala.analysis.AuthorizationTest.TestFunction[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropDatabase[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelSelect[1]
> org.apache.impala.analysis.AuthorizationTest.TestTruncate[1]
> org.apache.impala.analysis.AuthorizationTest.TestShowCreateTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestShortUsernameUsed[1]
> org.apache.impala.analysis.AuthorizationTest.TestComputeStatsTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestHs2GetColumns[1]
> org.apache.impala.analysis.AuthorizationTest.TestLoad[1]
> org.apache.impala.analysis.AuthorizationTest.TestCreateDatabase[1]
> org.apache.impala.analysis.AuthorizationTest.TestInsert[1]
> org.apache.impala.analysis.AuthorizationTest.TestSelect[1]
> org.apache.impala.analysis.AuthorizationTest.TestCreateView[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelAlter[1]
> org.apache.impala.analysis.AuthorizationTest.TestResetMetadata[1]
> org.apache.impala.analysis.AuthorizationTest.TestExplain[1]
> org.apache.impala.analysis.AuthorizationTest.TestCommentOn[1]
> org.apache.impala.analysis.AuthorizationTest.TestWithClause[1]
> org.apache.impala.analysis.AuthorizationTest.TestAlterView[1]
> org.apache.impala.analysis.AuthorizationTest.TestShowPermissions[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelDrop[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropView[1]
> org.apache.impala.analysis.AuthorizationTest.TestShowDbResultsFiltered[1]
>
> org.apache.impala.analysis.AuthorizationTest.TestShowTableResultsFiltered[1]
> org.apache.impala.analysis.AuthorizationTest.TestUnion[1]
> org.apache.impala.analysis.AuthorizationTest.TestUseDb[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropStats[1]
> org.apache.impala.analysis.AuthorizationTest.TestDropTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestHs2GetSchema[1]
> org.apache.impala.analysis.AuthorizationTest.TestHs2GetTables[1]
> org.apache.impala.analysis.AuthorizationTest.TestDescribeDb[1]
> org.apache.impala.analysis.AuthorizationTest.TestAlterTable[1]
> org.apache.impala.analysis.AuthorizationTest.TestDescribe[1]
> org.apache.impala.analysis.AuthorizationTest.TestServerLevelCreate[1]
> org.apache.impala.analysis.AuthorizationTest.TestCreateTable[1]
> {noformat}
> The errors are all along the lines of:
> {noformat}
> org.apache.impala.catalog.AuthorizationException: User 'jenkins' does not
> have privileges to execute 'INSERT' on: functional_parquet.alltypes
> at
> org.apache.impala.authorization.AuthorizationChecker.checkAccess(AuthorizationChecker.java:146)
> at
> org.apache.impala.analysis.AnalysisContext.authorizePrivilegeRequest(AnalysisContext.java:567)
> at
> org.apache.impala.analysis.AnalysisContext.authorize(AnalysisContext.java:530)
> at
> org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:409)
> at
> org.apache.impala.common.FrontendTestBase.parseAndAnalyze(FrontendTestBase.java:429)
> at
> org.apache.impala.analysis.AuthorizationTest.AuthzOk(AuthorizationTest.java:2841)
> at
> org.apache.impala.analysis.AuthorizationTest.AuthzOk(AuthorizationTest.java:2836)
> at
> org.apache.impala.analysis.AuthorizationTest.AuthzOk(AuthorizationTest.java:2832)
> at
>
[jira] [Reopened] (IMPALA-7111) ASAN heap-use-after-free in impala::HdfsPluginTextScanner::CheckPluginEnabled
[
https://issues.apache.org/jira/browse/IMPALA-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reopened IMPALA-7111:
---
Looks like we hit this on a non-ASAN build:
{noformat}
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2461/testReport/junit/data_errors.test_data_errors/TestHdfsScanNodeErrors/test_hdfs_scan_node_errors_exec_optionbatch_size___1___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___True___abort_on_error___1___debug_action___None___exec_single_node_rows_threshold___0table_format__text_none_/
{noformat}
> ASAN heap-use-after-free in impala::HdfsPluginTextScanner::CheckPluginEnabled
> -
>
> Key: IMPALA-7111
> URL: https://issues.apache.org/jira/browse/IMPALA-7111
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: asan, broken-build
>
> [~tarmstr...@cloudera.com] - I'm assigning this to you since you added this
> file in IMPALA-6941.
> {noformat}
> ==4582==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x603000e8aa28 at pc 0x017ab9b4 bp 0x7f67e5f6b650 sp 0x7f67e5f6b648
> READ of size 1 at 0x603000e8aa28 thread T9236
> #0 0x17ab9b3 in bool
> __gnu_cxx::__ops::_Iter_pred
> >::operator()<__gnu_cxx::__normal_iterator
> >(__gnu_cxx::__normal_iterator)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/predefined_ops.h:231:24
> #1 0x17ab745 in __gnu_cxx::__normal_iterator
> std::__find_if<__gnu_cxx::__normal_iterator,
> __gnu_cxx::__ops::_Iter_pred >
> >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator,
> __gnu_cxx::__ops::_Iter_pred >,
> std::random_access_iterator_tag)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/stl_algo.h:140:8
> #2 0x17ab2dc in __gnu_cxx::__normal_iterator
> std::__find_if<__gnu_cxx::__normal_iterator,
> __gnu_cxx::__ops::_Iter_pred >
> >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator,
> __gnu_cxx::__ops::_Iter_pred >)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/stl_algo.h:161:14
> #3 0x17aaf6c in __gnu_cxx::__normal_iterator
> std::find_if<__gnu_cxx::__normal_iterator,
> boost::algorithm::detail::is_any_ofF
> >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator,
> boost::algorithm::detail::is_any_ofF)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/stl_algo.h:3803:14
> #4 0x17aaba1 in boost::iterator_range<__gnu_cxx::__normal_iterator std::string> >
> boost::algorithm::detail::token_finderF
> >::operator()<__gnu_cxx::__normal_iterator
> >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator) const
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/detail/finder.hpp:565:41
> #5 0x17ac118 in
> boost::function2 std::string> >, __gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator
> >::operator()(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator) const
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #6 0x17abf8d in
> boost::algorithm::detail::find_iterator_base<__gnu_cxx::__normal_iterator std::string> >::do_find(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator) const
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/detail/find_iterator.hpp:63:32
> #7 0x17aa00c in
> boost::algorithm::split_iterator<__gnu_cxx::__normal_iterator std::string> >::increment()
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/find_iterator.hpp:305:44
> #8 0x17a95a5 in
> boost::algorithm::split_iterator<__gnu_cxx::__normal_iterator std::string>
> >::split_iterator
> > >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator,
> boost::algorithm::detail::token_finderF
> >)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/find_iterator.hpp:265:21
> #9 0x17a8d5e in std::vector >&
> boost::algorithm::iter_split std::allocator >, std::string,
> boost::algorithm::detail::token_finderF
> > >(std::vector >&, std::string&,
>
[jira] [Comment Edited] (IMPALA-7111) ASAN heap-use-after-free in impala::HdfsPluginTextScanner::CheckPluginEnabled
[
https://issues.apache.org/jira/browse/IMPALA-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508308#comment-16508308
]
Tim Armstrong edited comment on IMPALA-7111 at 6/11/18 4:24 PM:
Looks like we hit this on a non-ASAN build:
{noformat}
Error Message
data_errors/test_data_errors.py:124: in test_hdfs_scan_node_errors
self.run_test_case('DataErrorsTest/hdfs-scan-node-errors', vector)
common/impala_test_suite.py:405: in run_test_case result =
self.__execute_query(target_impalad_client, query, user=user)
common/impala_test_suite.py:620: in __execute_query return
impalad_client.execute(query, user=user) common/impala_connection.py:160: in
execute return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:173: in execute handle =
self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:341: in __execute_query
self.wait_for_completion(handle) beeswax/impala_beeswax.py:361: in
wait_for_completion raise ImpalaBeeswaxException("Query aborted:" +
error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: EQuery
aborted:Scanner plugin 'LZO' is not one of the enabled plugins: 'LZO'
Stacktrace
data_errors/test_data_errors.py:124: in test_hdfs_scan_node_errors
self.run_test_case('DataErrorsTest/hdfs-scan-node-errors', vector)
common/impala_test_suite.py:405: in run_test_case
result = self.__execute_query(target_impalad_client, query, user=user)
common/impala_test_suite.py:620: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:160: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:173: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:341: in __execute_query
self.wait_for_completion(handle)
beeswax/impala_beeswax.py:361: in wait_for_completion
raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
EQuery aborted:Scanner plugin 'LZO' is not one of the enabled plugins: 'LZO'
Standard Error
-- connecting to: localhost:21000
-- executing against localhost:21000
use functional;
SET batch_size=1;
SET num_nodes=0;
SET disable_codegen_rows_threshold=0;
SET disable_codegen=True;
SET abort_on_error=0;
SET exec_single_node_rows_threshold=0;
-- executing against localhost:21000
select id, bool_col, tinyint_col, smallint_col from alltypeserror order by id;
-- executing against localhost:21000
select count(*) from functional_text_lzo.bad_text_lzo;
{noformat}
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2461/testReport/junit/data_errors.test_data_errors/TestHdfsScanNodeErrors/test_hdfs_scan_node_errors_exec_optionbatch_size___1___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___True___abort_on_error___1___debug_action___None___exec_single_node_rows_threshold___0table_format__text_none_/
was (Author: tarmstrong):
Looks like we hit this on a non-ASAN build:
{noformat}
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2461/testReport/junit/data_errors.test_data_errors/TestHdfsScanNodeErrors/test_hdfs_scan_node_errors_exec_optionbatch_size___1___num_nodes___0___disable_codegen_rows_threshold___0___disable_codegen___True___abort_on_error___1___debug_action___None___exec_single_node_rows_threshold___0table_format__text_none_/
{noformat}
> ASAN heap-use-after-free in impala::HdfsPluginTextScanner::CheckPluginEnabled
> -
>
> Key: IMPALA-7111
> URL: https://issues.apache.org/jira/browse/IMPALA-7111
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: asan, broken-build
>
> [~tarmstr...@cloudera.com] - I'm assigning this to you since you added this
> file in IMPALA-6941.
> {noformat}
> ==4582==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x603000e8aa28 at pc 0x017ab9b4 bp 0x7f67e5f6b650 sp 0x7f67e5f6b648
> READ of size 1 at 0x603000e8aa28 thread T9236
> #0 0x17ab9b3 in bool
> __gnu_cxx::__ops::_Iter_pred
> >::operator()<__gnu_cxx::__normal_iterator
> >(__gnu_cxx::__normal_iterator)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/predefined_ops.h:231:24
> #1 0x17ab745 in __gnu_cxx::__normal_iterator
> std::__find_if<__gnu_cxx::__normal_iterator,
> __gnu_cxx::__ops::_Iter_pred >
> >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator,
> __gnu_cxx::__ops::_Iter_pred >,
>
[jira] [Resolved] (IMPALA-7078) Selective Avro scans of wide tables use more memory than necessary
[
https://issues.apache.org/jira/browse/IMPALA-7078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-7078.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
Impala 2.13.0
> Selective Avro scans of wide tables use more memory than necessary
> --
>
> Key: IMPALA-7078
> URL: https://issues.apache.org/jira/browse/IMPALA-7078
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.10.0, Impala 2.11.0, Impala 2.12.0, Impala
> 2.13.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Labels: avro, resource-management
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> We have some anecdotal evidence that this got worse somewhere between 2.7 and
> 2.10, but it may be because of a change in timing rather than a direct
> consequence of a code change. The scan uses ~500MB of memory, but should
> probably be running with less.
> Analysis revealed that most of the memory was from the tuple data buffers,
> which were multiple mb because they had space for 1024 wide rows.
> There are a couple of things we could do:
> * Directly improve Avro selective scans to avoid transferring excessive
> memory (i.e. something analogous to what IMPALA-4923 did for Parquet)
> * Tweak the row batch queueing to reduce unnecessary queueing. If the
> consumer is running slow, lots of batches pile up in the queue. We saw the
> regression on a system with 24 disks, so the queue was ~280 entries, which is
> excessive.
> Repro is as follows:
> {code}
> -- In Hive
> create table wide_250_cols (
> col0 STRING, col1 STRING, col2 STRING, col3 STRING, col4 STRING, col5
> STRING,
> col6 STRING, col7 STRING, col8 STRING, col9 STRING, col10 STRING, col11
> STRING,
> col12 STRING, col13 STRING, col14 STRING, col15 STRING, col16 STRING, col17
> STRING,
> col18 STRING, col19 STRING, col20 STRING, col21 STRING, col22 STRING, col23
> STRING,
> col24 STRING, col25 STRING, col26 STRING, col27 STRING, col28 STRING, col29
> STRING,
> col30 STRING, col31 STRING, col32 STRING, col33 STRING, col34 STRING, col35
> STRING,
> col36 STRING, col37 STRING, col38 STRING, col39 STRING, col40 STRING, col41
> STRING,
> col42 STRING, col43 STRING, col44 STRING, col45 STRING, col46 STRING, col47
> STRING,
> col48 STRING, col49 STRING, col50 STRING, col51 STRING, col52 STRING, col53
> STRING,
> col54 STRING, col55 STRING, col56 STRING, col57 STRING, col58 STRING, col59
> STRING,
> col60 STRING, col61 STRING, col62 STRING, col63 STRING, col64 STRING, col65
> STRING,
> col66 STRING, col67 STRING, col68 STRING, col69 STRING, col70 STRING, col71
> STRING,
> col72 STRING, col73 STRING, col74 STRING, col75 STRING, col76 STRING, col77
> STRING,
> col78 STRING, col79 STRING, col80 STRING, col81 STRING, col82 STRING, col83
> STRING,
> col84 STRING, col85 STRING, col86 STRING, col87 STRING, col88 STRING, col89
> STRING,
> col90 STRING, col91 STRING, col92 STRING, col93 STRING, col94 STRING, col95
> STRING,
> col96 STRING, col97 STRING, col98 STRING, col99 STRING, col100 STRING,
> col101 STRING,
> col102 STRING, col103 STRING, col104 STRING, col105 STRING, col106 STRING,
> col107 STRING,
> col108 STRING, col109 STRING, col110 STRING, col111 STRING, col112 STRING,
> col113 STRING,
> col114 STRING, col115 STRING, col116 STRING, col117 STRING, col118 STRING,
> col119 STRING,
> col120 STRING, col121 STRING, col122 STRING, col123 STRING, col124 STRING,
> col125 STRING,
> col126 STRING, col127 STRING, col128 STRING, col129 STRING, col130 STRING,
> col131 STRING,
> col132 STRING, col133 STRING, col134 STRING, col135 STRING, col136 STRING,
> col137 STRING,
> col138 STRING, col139 STRING, col140 STRING, col141 STRING, col142 STRING,
> col143 STRING,
> col144 STRING, col145 STRING, col146 STRING, col147 STRING, col148 STRING,
> col149 STRING,
> col150 STRING, col151 STRING, col152 STRING, col153 STRING, col154 STRING,
> col155 STRING,
> col156 STRING, col157 STRING, col158 STRING, col159 STRING, col160 STRING,
> col161 STRING,
> col162 STRING, col163 STRING, col164 STRING, col165 STRING, col166 STRING,
> col167 STRING,
> col168 STRING, col169 STRING, col170 STRING, col171 STRING, col172 STRING,
> col173 STRING,
> col174 STRING, col175 STRING, col176 STRING, col177 STRING, col178 STRING,
> col179 STRING,
> col180 STRING, col181 STRING, col182 STRING, col183 STRING, col184 STRING,
> col185 STRING,
> col186 STRING, col187 STRING, col188 STRING, col189 STRING, col190 STRING,
> col191 STRING,
> col192 STRING, col193 STRING, col194 STRING, col195 STRING, col196 STRING,
> col197 STRING,
> col198 STRING, col199 STRING, col200 STRING, col201
[jira] [Resolved] (IMPALA-411) Add testing for ReadWriteUtil
[ https://issues.apache.org/jira/browse/IMPALA-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-411. -- Resolution: Later This isn't a terrible idea but we don't need a JIRA to track adding more tests. > Add testing for ReadWriteUtil > - > > Key: IMPALA-411 > URL: https://issues.apache.org/jira/browse/IMPALA-411 > Project: IMPALA > Issue Type: Task > Components: Backend >Affects Versions: Impala 1.0 >Reporter: Nong Li >Priority: Minor > Labels: ramp-up > > This file contains some low level byte utilities. It is very unit testable > and we need a bit more coverage here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6942) "Cancelled due to unreachable impalad(s)" error message is misleading
[ https://issues.apache.org/jira/browse/IMPALA-6942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508962#comment-16508962 ] Tim Armstrong commented on IMPALA-6942: --- [~dhecht] this seems to be a one-word change. Any reason not to just do it now? > "Cancelled due to unreachable impalad(s)" error message is misleading > - > > Key: IMPALA-6942 > URL: https://issues.apache.org/jira/browse/IMPALA-6942 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.0 >Reporter: Dan Hecht >Priority: Major > > The error message "Cancelled due to unreachable impalad(s)" would be better > as "Failed due to unreachable impalad(s)" since the query has failed. The > code happens to trigger cancellation via the same path as client > cancellation, but this is really a query failure. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6160) tpcds-q22a.test breaks run-workload.py
[
https://issues.apache.org/jira/browse/IMPALA-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-6160:
-
Assignee: (was: Tim Wood)
> tpcds-q22a.test breaks run-workload.py
> --
>
> Key: IMPALA-6160
> URL: https://issues.apache.org/jira/browse/IMPALA-6160
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.11.0
>Reporter: Jim Apple
>Priority: Major
>
> From {{single_node_perf_run.py}}:
> {noformat}
> [scheduler] [tpcds20_parquet Thread 0]: Running Query: TPCDS-Q22A
> [query_exec_functions] [tpcds20_parquet Thread 0]: Connected to
> localhost:21001
> [query_exec_functions] [tpcds20_parquet Thread 0]: ImpalaBeeswaxException:
> INNER EXCEPTION:
> MESSAGE: AnalysisException: Syntax error in line 3:
> set decimal_v2=1;
> ^
> Encountered: SET
> Expected: CREATE, DELETE, INSERT, SELECT, UPDATE, UPSERT, VALUES, WITH
> CAUSED BY: Exception: Syntax error
> [query_exec_functions] [tpcds20_parquet Thread 0]: Connected to
> localhost:21001
> [query_exec_functions] [tpcds20_parquet Thread 0]: ImpalaBeeswaxException:
> INNER EXCEPTION:
> MESSAGE: AnalysisException: Syntax error in line 4:
> with results as
> ^
> Encountered: WITH
> Expected
> CAUSED BY: Exception: Syntax error
> {noformat}
> {noformat}
> Traceback (most recent call last):
> File "./bin/single_node_perf_run.py", line 314, in
> main()
> File "./bin/single_node_perf_run.py", line 304, in main
> perf_ab_test(options, args)
> File "./bin/single_node_perf_run.py", line 228, in perf_ab_test
> run_workload(temp_dir, workloads, options)
> File "./bin/single_node_perf_run.py", line 127, in run_workload
> subprocess.check_call(run_workload)
> File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
> raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command
> '['/home/ubuntu/Impala/bin/run-workload.py',
> '--workloads=tpch:20,targeted-perf:20,tpcds:20',
> '--impalads=localhost:21000,localhost:21001,localhost:21002',
> '--results_json_file=/home/ubuntu/Impala/perf_results/perf_run_d_73x1/6b37a793a0af74ba01c67bb5ac7285af7c5c4d52.json',
> '--query_iterations=50', '--table_formats=parquet/none', '--plan_first',
> '--query_names=.*']' returned non-zero exit status 1
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-6199) Flaky test: metadata/test_hdfs_permissions.py
[
https://issues.apache.org/jira/browse/IMPALA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-6199.
---
Resolution: Cannot Reproduce
> Flaky test: metadata/test_hdfs_permissions.py
> -
>
> Key: IMPALA-6199
> URL: https://issues.apache.org/jira/browse/IMPALA-6199
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.11.0
>Reporter: Taras Bobrovytsky
>Priority: Critical
>
> TestHdfsPermissions.test_insert_into_read_only_table failed on a nightly
> Isilon build with the following error message:
> {code}
> TestHdfsPermissions.test_insert_into_read_only_table[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
> 'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> [gw1] linux2 -- Python 2.6.6
> /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/bin/../infra/python/env/bin/python
> metadata/test_hdfs_permissions.py:73: in test_insert_into_read_only_table
> self.hdfs_client.delete_file_dir('test-warehouse/%s' % TEST_TBL,
> recursive=True)
> util/hdfs_util.py:90: in delete_file_dir
> if not self.exists(path):
> util/hdfs_util.py:138: in exists
> self.get_file_dir_status(path)
> util/hdfs_util.py:102: in get_file_dir_status
> return super(PyWebHdfsClientWithChmod, self).get_file_dir_status(path)
> ../infra/python/env/lib/python2.6/site-packages/pywebhdfs/webhdfs.py:338: in
> get_file_dir_status
> _raise_pywebhdfs_exception(response.status_code, response.content)
> ../infra/python/env/lib/python2.6/site-packages/pywebhdfs/webhdfs.py:477: in
> _raise_pywebhdfs_exception
> raise errors.PyWebHdfsException(msg=message)
> E PyWebHdfsException:
> E
> E 403 Forbidden
> E
> E Forbidden
> E You don't have permission to access /v1/html/500Text.html
> E on this server.
> E
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6074) All Build Options Jenkins job can fail when running on a previously-used node
[ https://issues.apache.org/jira/browse/IMPALA-6074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508932#comment-16508932 ] Tim Armstrong commented on IMPALA-6074: --- [~jbapple] I guess this hasn't reoccurred? > All Build Options Jenkins job can fail when running on a previously-used node > - > > Key: IMPALA-6074 > URL: https://issues.apache.org/jira/browse/IMPALA-6074 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Reporter: Jim Apple >Priority: Major > > See https://jenkins.impala.io/job/all-build-options-ub1604/16/console > Looks to be a problem with the git command sequence. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
