[jira] [Resolved] (IMPALA-7800) Reject or timeout new incoming connections once --fe_service_threads limit is reached
[
https://issues.apache.org/jira/browse/IMPALA-7800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga resolved IMPALA-7800.
--
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> Reject or timeout new incoming connections once --fe_service_threads limit is
> reached
> -
>
> Key: IMPALA-7800
> URL: https://issues.apache.org/jira/browse/IMPALA-7800
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Zoram Thanga
>Priority: Major
> Fix For: Impala 3.3.0
>
>
> Currently, the number of frontend service threads is controlled by
> {{--fe_service_threads}}. Once this limit is reached, the worker threads of
> {{connection_setup_pool}} will block until some of the service threads exit.
> New incoming connections will be buffered in a queue in
> {{connection_setup_pool}} until it fills up. Currently, there is no time out
> for all these buffered connections in the queue. Consequently, if the number
> of frontend service threads max'ed out for an extended period of time, a
> client trying to connect to Impala will feel "stuck" until a front end
> service thread is available.
> We should consider either rejecting any new connections once
> {{--fe_service_threads}} limit is reached or imposing a tunable upper bound
> on the wait time for connections in the queue to improve the users'
> experience. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-7508) Add Impala GDB commands to find fragment instances and queries in a core file
[
https://issues.apache.org/jira/browse/IMPALA-7508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga resolved IMPALA-7508.
--
Resolution: Fixed
Resolving now.
> Add Impala GDB commands to find fragment instances and queries in a core file
> -
>
> Key: IMPALA-7508
> URL: https://issues.apache.org/jira/browse/IMPALA-7508
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 2.12.0, Impala 3.1.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Minor
> Labels: debugging, supportability
>
> Introduce a Python GDB module, and a couple of commands that helps to find
> queries and fragment instances that are executing in an impalad at the time
> the daemon crashes.
> One hopes that folks will keep enhancing the module by adding new and useful
> GDB commands to aid debugging impalad core dumps.
> The initial patch has these:
> {noformat}
> (gdb) source impala-gdb.py
> (gdb) find-query-ids
> f74c863dff66a34d:1d983cc3
> 364525e12495932b:73f5dd02
> bc4a3eec25481981:edda04b8
> (gdb) find-fragment-instances
> Fragment Instance IdThread IDs
> 364525e12495932b:73f5dd0200a2 [69]
> 364525e12495932b:73f5dd020171 [196, 136]
> bc4a3eec25481981:edda04b801a8 [252, 237, 206]
> f74c863dff66a34d:1d983cc3009b [200, 14, 13, 12, 6, 5, 3, 2]
> f74c863dff66a34d:1d983cc3013a [4]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (IMPALA-889) Add trim() function matching ANSI SQL definition
[
https://issues.apache.org/jira/browse/IMPALA-889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga reassigned IMPALA-889:
---
Assignee: (was: Zoram Thanga)
> Add trim() function matching ANSI SQL definition
>
>
> Key: IMPALA-889
> URL: https://issues.apache.org/jira/browse/IMPALA-889
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend
>Affects Versions: Impala 1.2.4
>Reporter: Jonathan Seidman
>Priority: Minor
> Labels: ansi-sql, built-in-function, ramp-up
>
> Add support for an ISO-SQL compliant trim() function, i.e. trim([leading |
> trailing | both] [characters] from string). Lack of this impacts users of BI
> tools migrating existing SQL from other systems.
> Reference:
> https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/SQLReferenceManual/Functions/String/TRIM.htm
> Part of the ANSI definition
> {noformat}
> ::=
> TRIM
> ::=
> [ [ ] [ ] FROM ]
> ::=
>
> ::=
> LEADING
> | TRAILING
> | BOTH
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6665) Tag CatalogOp logs with query IDs
[ https://issues.apache.org/jira/browse/IMPALA-6665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga reassigned IMPALA-6665: Assignee: (was: Zoram Thanga) > Tag CatalogOp logs with query IDs > - > > Key: IMPALA-6665 > URL: https://issues.apache.org/jira/browse/IMPALA-6665 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Affects Versions: Impala 2.12.0 >Reporter: bharath v >Priority: Major > Labels: supportability > > Similar to IMPALA-6664. The idea is to improve catalog server logging by > adding query-ID to each of the Catalog server log statements. This helps map > Catalog errors to specific queries, which is currently not possible. > Raising a separate jira for the Catalog server since fixing it could be a > little tricker than the other components since we don't have the query hash > readily available in the Catalog context. We need to augment the Catalog RPCs > with this data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-5922) Scanners should include file and offset information in errors
[
https://issues.apache.org/jira/browse/IMPALA-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga reassigned IMPALA-5922:
Assignee: (was: Zoram Thanga)
> Scanners should include file and offset information in errors
> -
>
> Key: IMPALA-5922
> URL: https://issues.apache.org/jira/browse/IMPALA-5922
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Lars Volker
>Priority: Major
>
> Currently we have to print the location of a parse error in
> {{BaseSequenceScanner::GetNextInternal()}}:
> {code}
> state_->LogError(ErrorMsg(TErrorCode::SEQUENCE_SCANNER_PARSE_ERROR,
> stream_->filename(), stream_->file_offset(),
> (stream_->eof() ? "(EOF)" : "")));
> {code}
> Instead, the scanners should include this information when constructing the
> error, which will allow us to simplify the error handling in the base class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7674) Impala should compress older log files
[ https://issues.apache.org/jira/browse/IMPALA-7674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga reassigned IMPALA-7674: Assignee: (was: Zoram Thanga) > Impala should compress older log files > -- > > Key: IMPALA-7674 > URL: https://issues.apache.org/jira/browse/IMPALA-7674 > Project: IMPALA > Issue Type: Improvement >Affects Versions: Impala 3.0, Impala 2.12.0, Impala 3.1.0 >Reporter: Zoram Thanga >Priority: Major > Labels: supportability > > By default, Impala keeps ten log files of each severity level (INFO, WARN, > ERROR), and the size limit of each is set to 200MB or so. The cleaning or old > file deletion is controlled by the FLAGS_max_log_files parameter. > On busy clusters we've found that log deletion can throw away debug > information too quickly, often making troubleshooting harder than it needs to > be. > We can compress the log files to: > # Reduce the disk space consumption by 10x or more. > # Keep more log files around for the same disk space budget. > # Have 10x or more historical diagnostics data available. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8306) Debug WebUI's Sessions page verbiage clarification
[
https://issues.apache.org/jira/browse/IMPALA-8306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga reassigned IMPALA-8306:
Assignee: (was: Zoram Thanga)
> Debug WebUI's Sessions page verbiage clarification
> --
>
> Key: IMPALA-8306
> URL: https://issues.apache.org/jira/browse/IMPALA-8306
> Project: IMPALA
> Issue Type: Improvement
>Affects Versions: Impala 2.12.0, Impala 3.1.0
>Reporter: Vincent Tran
>Priority: Minor
> Attachments: sessions.png
>
>
> Currently, the Debug WebUI's Sessions page captures both active sessions and
> expired sessions. On the top of the page there is a message along the line of:
> {noformat}
> There are {{num_sessions}} sessions, of which {{num_active}} are active.
> Sessions may be closed either when they are idle for some time (see Idle
> Timeout
> below), or if they are deliberately closed, otherwise they are called active.
> {noformat}
> This text is ambiguous for me. If all non-active sessions are expired
> sessions, it should explicitly tell the user that. And since an active
> session becomes an expired session when it breaches the Session Idle Timeout,
> the second sentence is also somewhat misleading. User has to "deliberately
> close" both active sessions and expired sessions to close them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8310) Terminate impalad when HDFS call thread pool is depleted
[ https://issues.apache.org/jira/browse/IMPALA-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794014#comment-16794014 ] Zoram Thanga commented on IMPALA-8310: -- Ahh yes. Thanks [~joemcdonnell]. > Terminate impalad when HDFS call thread pool is depleted > > > Key: IMPALA-8310 > URL: https://issues.apache.org/jira/browse/IMPALA-8310 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.2.0 >Reporter: Lars Volker >Priority: Major > > JVM hangs will currently consume the thread pool added in IMPALA-7738. Once > the pool is depleted we abort new queries. Instead we should consider killing > the Impalad since it is in a state where it won’t be able to process queries > until it is restarted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8310) Terminate impalad when HDFS call thread pool is depleted
[ https://issues.apache.org/jira/browse/IMPALA-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792971#comment-16792971 ] Zoram Thanga commented on IMPALA-8310: -- Sounds good to me. That said, I wonder how the thread pool could be exhausted indefinitely. If the timeout actually works, shouldn't the threads be available once again after returning the timeout error status? > Terminate impalad when HDFS call thread pool is depleted > > > Key: IMPALA-8310 > URL: https://issues.apache.org/jira/browse/IMPALA-8310 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.2.0 >Reporter: Lars Volker >Priority: Major > > JVM hangs will currently consume the thread pool added in IMPALA-7738. Once > the pool is depleted we abort new queries. Instead we should consider killing > the Impalad since it is in a state where it won’t be able to process queries > until it is restarted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739902#comment-16739902
]
Zoram Thanga commented on IMPALA-7802:
--
Apologies, [~tarmstrong] for not responding sooner. I hadn't noticed your
comment until today.
It would be nice to have something like
[TThreadedSelectorServer|https://github.com/apache/thrift/blob/master/lib/java/src/org/apache/thrift/server/TThreadedSelectorServer.java],
but that would likely require too much work.
The best we can do might be to find a way to forcibly close the socket once all
sessions in the connection have timed out, as you suggest above. The tricky
thing we need to do is to change TAcceptQueueServer::Task::run()'s call to
TTransport->peek() so that it can be 'woken up' from the ImpalaServer layer.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Zoram Thanga
>Priority: Critical
> Labels: supportability
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7800) Reject or timeout new incoming connections once --fe_service_threads limit is reached
[
https://issues.apache.org/jira/browse/IMPALA-7800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739889#comment-16739889
]
Zoram Thanga commented on IMPALA-7800:
--
The current implementation of TAccepQueueServer inherits directly from TServer,
and essentially implements a blocking server, in the sense that once the accept
queue is full new connection requests are made to block. Also, accepted
connections also block in SetupConnection if maxTasks_ threads (controlled by
--fe_service_threads) are already in use. Since the accepted connection queue
is very big (10,000 by default) it's unlikely that we would bump against this
limit.
To enable us to reject OpenSession() requests on breaching the
fe_service_threads limit, we have a couple of options:
# Use the existing TAcceptQueueServer with a new (boolean)
blocking/non-blocking flag that we can check in
TAcceptQueueServer::SetupConnection(), and close/reject the connection once
we're at the threads limit.
# Implement a new, "non-blocking" TNonblockingServer which is basically a
non-blocking variant of TAcceptQueueServer, which can be instantiated instead
in ImpalaServer::Start() if a non-blocking thrift server is desired by the
startup flag (--use_nonblocking_thrift_server=true, or something like that).
The second option might be an overkill though. I've tried the first option and
it seems to work well enough in my limited testing.
Thoughts? cc: [~tarmstr...@cloudera.com], [~kwho]
> Reject or timeout new incoming connections once --fe_service_threads limit is
> reached
> -
>
> Key: IMPALA-7800
> URL: https://issues.apache.org/jira/browse/IMPALA-7800
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Zoram Thanga
>Priority: Major
>
> Currently, the number of frontend service threads is controlled by
> {{--fe_service_threads}}. Once this limit is reached, the worker threads of
> {{connection_setup_pool}} will block until some of the service threads exit.
> New incoming connections will be buffered in a queue in
> {{connection_setup_pool}} until it fills up. Currently, there is no time out
> for all these buffered connections in the queue. Consequently, if the number
> of frontend service threads max'ed out for an extended period of time, a
> client trying to connect to Impala will feel "stuck" until a front end
> service thread is available.
> We should consider either rejecting any new connections once
> {{--fe_service_threads}} limit is reached or imposing a tunable upper bound
> on the wait time for connections in the queue to improve the users'
> experience. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7800) Reject or timeout new incoming connections once --fe_service_threads limit is reached
[
https://issues.apache.org/jira/browse/IMPALA-7800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga reassigned IMPALA-7800:
Assignee: Zoram Thanga
> Reject or timeout new incoming connections once --fe_service_threads limit is
> reached
> -
>
> Key: IMPALA-7800
> URL: https://issues.apache.org/jira/browse/IMPALA-7800
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Zoram Thanga
>Priority: Major
>
> Currently, the number of frontend service threads is controlled by
> {{--fe_service_threads}}. Once this limit is reached, the worker threads of
> {{connection_setup_pool}} will block until some of the service threads exit.
> New incoming connections will be buffered in a queue in
> {{connection_setup_pool}} until it fills up. Currently, there is no time out
> for all these buffered connections in the queue. Consequently, if the number
> of frontend service threads max'ed out for an extended period of time, a
> client trying to connect to Impala will feel "stuck" until a front end
> service thread is available.
> We should consider either rejecting any new connections once
> {{--fe_service_threads}} limit is reached or imposing a tunable upper bound
> on the wait time for connections in the queue to improve the users'
> experience. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6664) Tag log statements with query-ids
[ https://issues.apache.org/jira/browse/IMPALA-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga reassigned IMPALA-6664: Assignee: Philip Zeyliger (was: Zoram Thanga) > Tag log statements with query-ids > - > > Key: IMPALA-6664 > URL: https://issues.apache.org/jira/browse/IMPALA-6664 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Affects Versions: Impala 2.12.0 >Reporter: bharath v >Assignee: Philip Zeyliger >Priority: Major > Labels: supportability > > The idea is to include query-id in each logged statement in > coordinator/finsts/planner. This helps to collect query specific logs by just > grep'ing through the logs with the query-id. > This is hopefully easier to implement, now that we have the query debug info > available in the thread context [1]. > [1] https://issues.apache.org/jira/browse/IMPALA-3703 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8012) Log a message when --fe_service_threads have been allocated
Zoram Thanga created IMPALA-8012: Summary: Log a message when --fe_service_threads have been allocated Key: IMPALA-8012 URL: https://issues.apache.org/jira/browse/IMPALA-8012 Project: IMPALA Issue Type: Improvement Components: Backend, Clients Reporter: Zoram Thanga The maximum number of front end service threads that can be created at any time to handle client connections is controlled by the "--fe_service_threads" parameter. When all such threads have been allocated, new connection requests get queued, and in theory can be spending an indefinite amount of time in the queue. Users perceive this as slow impala connection setup time. We should log a message when we're near or at --fe_service_threads threads to make debugging this situation easier. cc: [~kwho] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8012) Log a message when --fe_service_threads have been allocated
Zoram Thanga created IMPALA-8012: Summary: Log a message when --fe_service_threads have been allocated Key: IMPALA-8012 URL: https://issues.apache.org/jira/browse/IMPALA-8012 Project: IMPALA Issue Type: Improvement Components: Backend, Clients Reporter: Zoram Thanga The maximum number of front end service threads that can be created at any time to handle client connections is controlled by the "--fe_service_threads" parameter. When all such threads have been allocated, new connection requests get queued, and in theory can be spending an indefinite amount of time in the queue. Users perceive this as slow impala connection setup time. We should log a message when we're near or at --fe_service_threads threads to make debugging this situation easier. cc: [~kwho] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718239#comment-16718239
]
Zoram Thanga commented on IMPALA-7802:
--
Thanks for the comment, [~tarmstrong]. You brought up a good point regarding
what the client experience has to be, when the session has expired. I think we
will have to maintain the current behavior, but without tying up FE service
threads. That is, expired sessions should not consume any resources on the
Impala server besides the session states.
I am thinking about moving expired sessions to a separate list (a death row)
where they remain until explicitly cancelled and/or closed by the client.
Perhaps we can have a single thread handle cancellation and closing of expired
sessions.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Zoram Thanga
>Priority: Critical
> Labels: supportability
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710704#comment-16710704
]
Zoram Thanga commented on IMPALA-7802:
--
The documentation states that:
{quote}
Once a session is expired, you cannot issue any new query requests to it. The
session remains open, but the only operation you can perform is to close it.
{quote}
This basically says that an expired session serves no useful purpose to any one
- not to Impala as it consumes an fe_service_thread, and not to the client
because the only operation allowed on it is to close it.
I would like to change the session expiry code to always force-close expired
sessions from the server side by calling ImpalaServer::CloseSessionInternal()
or a modified version of it.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Zoram Thanga
>Priority: Critical
> Labels: supportability
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-7802 started by Zoram Thanga.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Zoram Thanga
>Priority: Critical
> Labels: supportability
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga reassigned IMPALA-7802:
Assignee: Zoram Thanga
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Zoram Thanga
>Priority: Critical
> Labels: supportability
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga updated IMPALA-7802:
-
Labels: supportability (was: )
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Priority: Critical
> Labels: supportability
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7674) Impala should compress older log files
[ https://issues.apache.org/jira/browse/IMPALA-7674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga updated IMPALA-7674: - Labels: supportability (was: ) > Impala should compress older log files > -- > > Key: IMPALA-7674 > URL: https://issues.apache.org/jira/browse/IMPALA-7674 > Project: IMPALA > Issue Type: Improvement >Affects Versions: Impala 3.0, Impala 2.12.0, Impala 3.1.0 >Reporter: Zoram Thanga >Assignee: Zoram Thanga >Priority: Major > Labels: supportability > > By default, Impala keeps ten log files of each severity level (INFO, WARN, > ERROR), and the size limit of each is set to 200MB or so. The cleaning or old > file deletion is controlled by the FLAGS_max_log_files parameter. > On busy clusters we've found that log deletion can throw away debug > information too quickly, often making troubleshooting harder than it needs to > be. > We can compress the log files to: > # Reduce the disk space consumption by 10x or more. > # Keep more log files around for the same disk space budget. > # Have 10x or more historical diagnostics data available. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7674) Impala should compress older log files
Zoram Thanga created IMPALA-7674: Summary: Impala should compress older log files Key: IMPALA-7674 URL: https://issues.apache.org/jira/browse/IMPALA-7674 Project: IMPALA Issue Type: Improvement Affects Versions: Impala 2.12.0, Impala 3.0, Impala 3.1.0 Reporter: Zoram Thanga Assignee: Zoram Thanga By default, Impala keeps ten log files of each severity level (INFO, WARN, ERROR), and the size limit of each is set to 200MB or so. The cleaning or old file deletion is controlled by the FLAGS_max_log_files parameter. On busy clusters we've found that log deletion can throw away debug information too quickly, often making troubleshooting harder than it needs to be. We can compress the log files to: # Reduce the disk space consumption by 10x or more. # Keep more log files around for the same disk space budget. # Have 10x or more historical diagnostics data available. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-7674) Impala should compress older log files
Zoram Thanga created IMPALA-7674: Summary: Impala should compress older log files Key: IMPALA-7674 URL: https://issues.apache.org/jira/browse/IMPALA-7674 Project: IMPALA Issue Type: Improvement Affects Versions: Impala 2.12.0, Impala 3.0, Impala 3.1.0 Reporter: Zoram Thanga Assignee: Zoram Thanga By default, Impala keeps ten log files of each severity level (INFO, WARN, ERROR), and the size limit of each is set to 200MB or so. The cleaning or old file deletion is controlled by the FLAGS_max_log_files parameter. On busy clusters we've found that log deletion can throw away debug information too quickly, often making troubleshooting harder than it needs to be. We can compress the log files to: # Reduce the disk space consumption by 10x or more. # Keep more log files around for the same disk space budget. # Have 10x or more historical diagnostics data available. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-6764) Codegend UnionNode::MaterializeBatch() causes memory corruption crash of Impalad
[
https://issues.apache.org/jira/browse/IMPALA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga updated IMPALA-6764:
-
Fix Version/s: Impala 2.11.0
> Codegend UnionNode::MaterializeBatch() causes memory corruption crash of
> Impalad
>
>
> Key: IMPALA-6764
> URL: https://issues.apache.org/jira/browse/IMPALA-6764
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Critical
> Fix For: Impala 2.11.0, Impala 2.12.0
>
> Attachments: bad-materializebatch-disasm.txt,
> good-materializebatch-disasm.txt
>
>
> A CTAS statement involving UNION ALL with LEFT JOIN children is reliably
> crashing with a stack trace similar to the following:
> {noformat}
> (gdb) bt
> #0 0x7fb85fdf11f7 in raise () from ./debug-stuff/lib64/libc.so.6
> #1 0x7fb85fdf28e8 in abort () from ./debug-stuff/lib64/libc.so.6
> #2 0x7fb862106f35 in os::abort(bool) () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #3 0x7fb8622aaf33 in VMError::report_and_die() () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #4 0x7fb86210d22f in JVM_handle_linux_signal () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #5 0x7fb862103253 in signalHandler(int, siginfo*, void*) () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x7fb85ff08706 in __memcpy_ssse3_back () from
> ./debug-stuff/lib64/libc.so.6
> #8 0x7fb840700d73 in
> impala::UnionNode::MaterializeBatch(impala::RowBatch*, unsigned char**)
> [clone .588] ()
> #9 0x01001806 in impala::UnionNode::GetNextMaterialized
> (this=this@entry=0x828, state=state@entry=0x848ed00,
> row_batch=row_batch@entry=0xcef9950)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:228
> #10 0x01001b5c in impala::UnionNode::GetNext (this=0x828,
> state=0x848ed00, row_batch=0xcef9950, eos=0x7fb7fe9a987e)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:294
> #11 0x00b724d2 in impala::FragmentInstanceState::ExecInternal
> (this=this@entry=0x4c030c0)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:270
> #12 0x00b74e42 in impala::FragmentInstanceState::Exec
> (this=this@entry=0x4c030c0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:89
> #13 0x00b64488 in impala::QueryState::ExecFInstance (this=0x8559200,
> fis=0x4c030c0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/query-state.cc:382
> #14 0x00d13613 in boost::function0::operator()
> (this=0x7fb7fe9a9c60)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #15 impala::Thread::SuperviseThread(std::string const&, std::string const&,
> boost::function, impala::Promise*) (name=..., category=...,
> functor=...,
> thread_started=0x7fb7f999f0f0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/util/thread.cc:352
> #16 0x00d13d54 in
> boost::_bi::list4 std::char_traits, std::allocator > >,
> boost::_bi::value,
> std::allocator > >, boost::_bi::value >,
> boost::_bi::value*> >::operator() std::basic_string&, const std::basic_string&,
> boost::function, impala::Promise*), boost::_bi::list0> (
> f=@0x808bfb8: 0xd13460 const&, std::string const&, boost::function,
> impala::Promise*)>, a=,
> this=0x808bfc0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
> #17 boost::_bi::bind_t boost::function, impala::Promise*),
> boost::_bi::list4,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value*> > >::operator()() (this=0x808bfb8)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
> #18 boost::detail::thread_data const&, std::string const&, boost::function,
> impala::Promise*), boost::_bi::list4,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value*> > > >::run() (this=0x808be00)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
> #19 0x0128e8ea in thread_proxy ()
> #20 0x7fb860186e25 in start_thread () from
> ./debug-stuff/lib64/libpthread.so.0
> #21 0x7fb85feb434d in clone () from ./debug-stuff/lib64/libc.so.6
> {noformat}
> The exact location or reason of the crash varies, i.e., sometimes we crash
> while accessing the source address of memcpy, other times we crash on
> accessing the destination address. In this
[jira] [Resolved] (IMPALA-6764) Codegend UnionNode::MaterializeBatch() causes memory corruption crash of Impalad
[
https://issues.apache.org/jira/browse/IMPALA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga resolved IMPALA-6764.
--
Resolution: Fixed
Fix Version/s: Impala 2.12.0
> Codegend UnionNode::MaterializeBatch() causes memory corruption crash of
> Impalad
>
>
> Key: IMPALA-6764
> URL: https://issues.apache.org/jira/browse/IMPALA-6764
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Critical
> Fix For: Impala 2.12.0
>
> Attachments: bad-materializebatch-disasm.txt,
> good-materializebatch-disasm.txt
>
>
> A CTAS statement involving UNION ALL with LEFT JOIN children is reliably
> crashing with a stack trace similar to the following:
> {noformat}
> (gdb) bt
> #0 0x7fb85fdf11f7 in raise () from ./debug-stuff/lib64/libc.so.6
> #1 0x7fb85fdf28e8 in abort () from ./debug-stuff/lib64/libc.so.6
> #2 0x7fb862106f35 in os::abort(bool) () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #3 0x7fb8622aaf33 in VMError::report_and_die() () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #4 0x7fb86210d22f in JVM_handle_linux_signal () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #5 0x7fb862103253 in signalHandler(int, siginfo*, void*) () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x7fb85ff08706 in __memcpy_ssse3_back () from
> ./debug-stuff/lib64/libc.so.6
> #8 0x7fb840700d73 in
> impala::UnionNode::MaterializeBatch(impala::RowBatch*, unsigned char**)
> [clone .588] ()
> #9 0x01001806 in impala::UnionNode::GetNextMaterialized
> (this=this@entry=0x828, state=state@entry=0x848ed00,
> row_batch=row_batch@entry=0xcef9950)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:228
> #10 0x01001b5c in impala::UnionNode::GetNext (this=0x828,
> state=0x848ed00, row_batch=0xcef9950, eos=0x7fb7fe9a987e)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:294
> #11 0x00b724d2 in impala::FragmentInstanceState::ExecInternal
> (this=this@entry=0x4c030c0)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:270
> #12 0x00b74e42 in impala::FragmentInstanceState::Exec
> (this=this@entry=0x4c030c0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:89
> #13 0x00b64488 in impala::QueryState::ExecFInstance (this=0x8559200,
> fis=0x4c030c0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/query-state.cc:382
> #14 0x00d13613 in boost::function0::operator()
> (this=0x7fb7fe9a9c60)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #15 impala::Thread::SuperviseThread(std::string const&, std::string const&,
> boost::function, impala::Promise*) (name=..., category=...,
> functor=...,
> thread_started=0x7fb7f999f0f0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/util/thread.cc:352
> #16 0x00d13d54 in
> boost::_bi::list4 std::char_traits, std::allocator > >,
> boost::_bi::value,
> std::allocator > >, boost::_bi::value >,
> boost::_bi::value*> >::operator() std::basic_string&, const std::basic_string&,
> boost::function, impala::Promise*), boost::_bi::list0> (
> f=@0x808bfb8: 0xd13460 const&, std::string const&, boost::function,
> impala::Promise*)>, a=,
> this=0x808bfc0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
> #17 boost::_bi::bind_t boost::function, impala::Promise*),
> boost::_bi::list4,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value*> > >::operator()() (this=0x808bfb8)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
> #18 boost::detail::thread_data const&, std::string const&, boost::function,
> impala::Promise*), boost::_bi::list4,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value*> > > >::run() (this=0x808be00)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
> #19 0x0128e8ea in thread_proxy ()
> #20 0x7fb860186e25 in start_thread () from
> ./debug-stuff/lib64/libpthread.so.0
> #21 0x7fb85feb434d in clone () from ./debug-stuff/lib64/libc.so.6
> {noformat}
> The exact location or reason of the crash varies, i.e., sometimes we crash
> while accessing the source address of memcpy, other times we crash on
> accessing the destination
[jira] [Resolved] (IMPALA-6764) Codegend UnionNode::MaterializeBatch() causes memory corruption crash of Impalad
[
https://issues.apache.org/jira/browse/IMPALA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga resolved IMPALA-6764.
--
Resolution: Fixed
Fix Version/s: Impala 2.12.0
> Codegend UnionNode::MaterializeBatch() causes memory corruption crash of
> Impalad
>
>
> Key: IMPALA-6764
> URL: https://issues.apache.org/jira/browse/IMPALA-6764
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Critical
> Fix For: Impala 2.12.0
>
> Attachments: bad-materializebatch-disasm.txt,
> good-materializebatch-disasm.txt
>
>
> A CTAS statement involving UNION ALL with LEFT JOIN children is reliably
> crashing with a stack trace similar to the following:
> {noformat}
> (gdb) bt
> #0 0x7fb85fdf11f7 in raise () from ./debug-stuff/lib64/libc.so.6
> #1 0x7fb85fdf28e8 in abort () from ./debug-stuff/lib64/libc.so.6
> #2 0x7fb862106f35 in os::abort(bool) () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #3 0x7fb8622aaf33 in VMError::report_and_die() () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #4 0x7fb86210d22f in JVM_handle_linux_signal () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #5 0x7fb862103253 in signalHandler(int, siginfo*, void*) () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x7fb85ff08706 in __memcpy_ssse3_back () from
> ./debug-stuff/lib64/libc.so.6
> #8 0x7fb840700d73 in
> impala::UnionNode::MaterializeBatch(impala::RowBatch*, unsigned char**)
> [clone .588] ()
> #9 0x01001806 in impala::UnionNode::GetNextMaterialized
> (this=this@entry=0x828, state=state@entry=0x848ed00,
> row_batch=row_batch@entry=0xcef9950)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:228
> #10 0x01001b5c in impala::UnionNode::GetNext (this=0x828,
> state=0x848ed00, row_batch=0xcef9950, eos=0x7fb7fe9a987e)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:294
> #11 0x00b724d2 in impala::FragmentInstanceState::ExecInternal
> (this=this@entry=0x4c030c0)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:270
> #12 0x00b74e42 in impala::FragmentInstanceState::Exec
> (this=this@entry=0x4c030c0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:89
> #13 0x00b64488 in impala::QueryState::ExecFInstance (this=0x8559200,
> fis=0x4c030c0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/query-state.cc:382
> #14 0x00d13613 in boost::function0::operator()
> (this=0x7fb7fe9a9c60)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #15 impala::Thread::SuperviseThread(std::string const&, std::string const&,
> boost::function, impala::Promise*) (name=..., category=...,
> functor=...,
> thread_started=0x7fb7f999f0f0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/util/thread.cc:352
> #16 0x00d13d54 in
> boost::_bi::list4 std::char_traits, std::allocator > >,
> boost::_bi::value,
> std::allocator > >, boost::_bi::value >,
> boost::_bi::value*> >::operator() std::basic_string&, const std::basic_string&,
> boost::function, impala::Promise*), boost::_bi::list0> (
> f=@0x808bfb8: 0xd13460 const&, std::string const&, boost::function,
> impala::Promise*)>, a=,
> this=0x808bfc0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
> #17 boost::_bi::bind_t boost::function, impala::Promise*),
> boost::_bi::list4,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value*> > >::operator()() (this=0x808bfb8)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
> #18 boost::detail::thread_data const&, std::string const&, boost::function,
> impala::Promise*), boost::_bi::list4,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value*> > > >::run() (this=0x808be00)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
> #19 0x0128e8ea in thread_proxy ()
> #20 0x7fb860186e25 in start_thread () from
> ./debug-stuff/lib64/libpthread.so.0
> #21 0x7fb85feb434d in clone () from ./debug-stuff/lib64/libc.so.6
> {noformat}
> The exact location or reason of the crash varies, i.e., sometimes we crash
> while accessing the source address of memcpy, other times we crash on
> accessing the destination
[jira] [Commented] (IMPALA-6764) Codegend UnionNode::MaterializeBatch() causes memory corruption crash of Impalad
[
https://issues.apache.org/jira/browse/IMPALA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604907#comment-16604907
]
Zoram Thanga commented on IMPALA-6764:
--
IMPALA-6059 changed the implementation of StringFunctions::Trim() enough that
this is not reproducible anymore.
Marking this as fixed by the above-mentioned JIRA.
> Codegend UnionNode::MaterializeBatch() causes memory corruption crash of
> Impalad
>
>
> Key: IMPALA-6764
> URL: https://issues.apache.org/jira/browse/IMPALA-6764
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Critical
> Attachments: bad-materializebatch-disasm.txt,
> good-materializebatch-disasm.txt
>
>
> A CTAS statement involving UNION ALL with LEFT JOIN children is reliably
> crashing with a stack trace similar to the following:
> {noformat}
> (gdb) bt
> #0 0x7fb85fdf11f7 in raise () from ./debug-stuff/lib64/libc.so.6
> #1 0x7fb85fdf28e8 in abort () from ./debug-stuff/lib64/libc.so.6
> #2 0x7fb862106f35 in os::abort(bool) () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #3 0x7fb8622aaf33 in VMError::report_and_die() () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #4 0x7fb86210d22f in JVM_handle_linux_signal () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #5 0x7fb862103253 in signalHandler(int, siginfo*, void*) () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x7fb85ff08706 in __memcpy_ssse3_back () from
> ./debug-stuff/lib64/libc.so.6
> #8 0x7fb840700d73 in
> impala::UnionNode::MaterializeBatch(impala::RowBatch*, unsigned char**)
> [clone .588] ()
> #9 0x01001806 in impala::UnionNode::GetNextMaterialized
> (this=this@entry=0x828, state=state@entry=0x848ed00,
> row_batch=row_batch@entry=0xcef9950)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:228
> #10 0x01001b5c in impala::UnionNode::GetNext (this=0x828,
> state=0x848ed00, row_batch=0xcef9950, eos=0x7fb7fe9a987e)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:294
> #11 0x00b724d2 in impala::FragmentInstanceState::ExecInternal
> (this=this@entry=0x4c030c0)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:270
> #12 0x00b74e42 in impala::FragmentInstanceState::Exec
> (this=this@entry=0x4c030c0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:89
> #13 0x00b64488 in impala::QueryState::ExecFInstance (this=0x8559200,
> fis=0x4c030c0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/query-state.cc:382
> #14 0x00d13613 in boost::function0::operator()
> (this=0x7fb7fe9a9c60)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #15 impala::Thread::SuperviseThread(std::string const&, std::string const&,
> boost::function, impala::Promise*) (name=..., category=...,
> functor=...,
> thread_started=0x7fb7f999f0f0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/util/thread.cc:352
> #16 0x00d13d54 in
> boost::_bi::list4 std::char_traits, std::allocator > >,
> boost::_bi::value,
> std::allocator > >, boost::_bi::value >,
> boost::_bi::value*> >::operator() std::basic_string&, const std::basic_string&,
> boost::function, impala::Promise*), boost::_bi::list0> (
> f=@0x808bfb8: 0xd13460 const&, std::string const&, boost::function,
> impala::Promise*)>, a=,
> this=0x808bfc0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
> #17 boost::_bi::bind_t boost::function, impala::Promise*),
> boost::_bi::list4,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value*> > >::operator()() (this=0x808bfb8)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
> #18 boost::detail::thread_data const&, std::string const&, boost::function,
> impala::Promise*), boost::_bi::list4,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value*> > > >::run() (this=0x808be00)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
> #19 0x0128e8ea in thread_proxy ()
> #20 0x7fb860186e25 in start_thread () from
> ./debug-stuff/lib64/libpthread.so.0
> #21 0x7fb85feb434d in clone () from ./debug-stuff/lib64/libc.so.6
> {noformat}
> The exact location or reason of the crash varies, i.e., sometimes we crash
[jira] [Updated] (IMPALA-7508) Add Impala GDB commands to find fragment instances and queries in a core file
[
https://issues.apache.org/jira/browse/IMPALA-7508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga updated IMPALA-7508:
-
Labels: debugging supportability (was: supportability)
> Add Impala GDB commands to find fragment instances and queries in a core file
> -
>
> Key: IMPALA-7508
> URL: https://issues.apache.org/jira/browse/IMPALA-7508
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 2.12.0, Impala 3.1.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Minor
> Labels: debugging, supportability
>
> Introduce a Python GDB module, and a couple of commands that helps to find
> queries and fragment instances that are executing in an impalad at the time
> the daemon crashes.
> One hopes that folks will keep enhancing the module by adding new and useful
> GDB commands to aid debugging impalad core dumps.
> The initial patch has these:
> {noformat}
> (gdb) source impala-gdb.py
> (gdb) find-query-ids
> f74c863dff66a34d:1d983cc3
> 364525e12495932b:73f5dd02
> bc4a3eec25481981:edda04b8
> (gdb) find-fragment-instances
> Fragment Instance IdThread IDs
> 364525e12495932b:73f5dd0200a2 [69]
> 364525e12495932b:73f5dd020171 [196, 136]
> bc4a3eec25481981:edda04b801a8 [252, 237, 206]
> f74c863dff66a34d:1d983cc3009b [200, 14, 13, 12, 6, 5, 3, 2]
> f74c863dff66a34d:1d983cc3013a [4]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7508) Add Impala GDB commands to find fragment instances and queries in a core file
[
https://issues.apache.org/jira/browse/IMPALA-7508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga updated IMPALA-7508:
-
Labels: supportability (was: )
> Add Impala GDB commands to find fragment instances and queries in a core file
> -
>
> Key: IMPALA-7508
> URL: https://issues.apache.org/jira/browse/IMPALA-7508
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 2.12.0, Impala 3.1.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Minor
> Labels: supportability
>
> Introduce a Python GDB module, and a couple of commands that helps to find
> queries and fragment instances that are executing in an impalad at the time
> the daemon crashes.
> One hopes that folks will keep enhancing the module by adding new and useful
> GDB commands to aid debugging impalad core dumps.
> The initial patch has these:
> {noformat}
> (gdb) source impala-gdb.py
> (gdb) find-query-ids
> f74c863dff66a34d:1d983cc3
> 364525e12495932b:73f5dd02
> bc4a3eec25481981:edda04b8
> (gdb) find-fragment-instances
> Fragment Instance IdThread IDs
> 364525e12495932b:73f5dd0200a2 [69]
> 364525e12495932b:73f5dd020171 [196, 136]
> bc4a3eec25481981:edda04b801a8 [252, 237, 206]
> f74c863dff66a34d:1d983cc3009b [200, 14, 13, 12, 6, 5, 3, 2]
> f74c863dff66a34d:1d983cc3013a [4]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7508) Add Impala GDB commands to find fragment instances and queries in a core file
Zoram Thanga created IMPALA-7508:
Summary: Add Impala GDB commands to find fragment instances and
queries in a core file
Key: IMPALA-7508
URL: https://issues.apache.org/jira/browse/IMPALA-7508
Project: IMPALA
Issue Type: Improvement
Components: Infrastructure
Affects Versions: Impala 2.12.0, Impala 3.1.0
Reporter: Zoram Thanga
Assignee: Zoram Thanga
Introduce a Python GDB module, and a couple of commands that helps to find
queries and fragment instances that are executing in an impalad at the time the
daemon crashes.
One hopes that folks will keep enhancing the module by adding new and useful
GDB commands to aid debugging impalad core dumps.
The initial patch has these:
{noformat}
(gdb) source impala-gdb.py
(gdb) find-query-ids
f74c863dff66a34d:1d983cc3
364525e12495932b:73f5dd02
bc4a3eec25481981:edda04b8
(gdb) find-fragment-instances
Fragment Instance IdThread IDs
364525e12495932b:73f5dd0200a2 [69]
364525e12495932b:73f5dd020171 [196, 136]
bc4a3eec25481981:edda04b801a8 [252, 237, 206]
f74c863dff66a34d:1d983cc3009b [200, 14, 13, 12, 6, 5, 3, 2]
f74c863dff66a34d:1d983cc3013a [4]
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7508) Add Impala GDB commands to find fragment instances and queries in a core file
[
https://issues.apache.org/jira/browse/IMPALA-7508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-7508 started by Zoram Thanga.
> Add Impala GDB commands to find fragment instances and queries in a core file
> -
>
> Key: IMPALA-7508
> URL: https://issues.apache.org/jira/browse/IMPALA-7508
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 2.12.0, Impala 3.1.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Minor
>
> Introduce a Python GDB module, and a couple of commands that helps to find
> queries and fragment instances that are executing in an impalad at the time
> the daemon crashes.
> One hopes that folks will keep enhancing the module by adding new and useful
> GDB commands to aid debugging impalad core dumps.
> The initial patch has these:
> {noformat}
> (gdb) source impala-gdb.py
> (gdb) find-query-ids
> f74c863dff66a34d:1d983cc3
> 364525e12495932b:73f5dd02
> bc4a3eec25481981:edda04b8
> (gdb) find-fragment-instances
> Fragment Instance IdThread IDs
> 364525e12495932b:73f5dd0200a2 [69]
> 364525e12495932b:73f5dd020171 [196, 136]
> bc4a3eec25481981:edda04b801a8 [252, 237, 206]
> f74c863dff66a34d:1d983cc3009b [200, 14, 13, 12, 6, 5, 3, 2]
> f74c863dff66a34d:1d983cc3013a [4]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7444) Improve debug logging of session opening and expiry
[ https://issues.apache.org/jira/browse/IMPALA-7444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga resolved IMPALA-7444. -- Resolution: Fixed Fix Version/s: Impala 3.1.0 > Improve debug logging of session opening and expiry > --- > > Key: IMPALA-7444 > URL: https://issues.apache.org/jira/browse/IMPALA-7444 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.0, Impala 2.12.0, Impala 3.1.0 >Reporter: Zoram Thanga >Assignee: Zoram Thanga >Priority: Minor > Labels: supportability > Fix For: Impala 3.1.0 > > > I recently had to troubleshoot JDBC client connection slowness, and looking > through this part of the code suggests we could improve the logging to make > debugging this kind of issue easier. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-7444) Improve debug logging of session opening and expiry
[ https://issues.apache.org/jira/browse/IMPALA-7444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga resolved IMPALA-7444. -- Resolution: Fixed Fix Version/s: Impala 3.1.0 > Improve debug logging of session opening and expiry > --- > > Key: IMPALA-7444 > URL: https://issues.apache.org/jira/browse/IMPALA-7444 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.0, Impala 2.12.0, Impala 3.1.0 >Reporter: Zoram Thanga >Assignee: Zoram Thanga >Priority: Minor > Labels: supportability > Fix For: Impala 3.1.0 > > > I recently had to troubleshoot JDBC client connection slowness, and looking > through this part of the code suggests we could improve the logging to make > debugging this kind of issue easier. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7444) Improve debug logging of session opening and expiry
[ https://issues.apache.org/jira/browse/IMPALA-7444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-7444 started by Zoram Thanga. > Improve debug logging of session opening and expiry > --- > > Key: IMPALA-7444 > URL: https://issues.apache.org/jira/browse/IMPALA-7444 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.0, Impala 2.12.0, Impala 3.1.0 >Reporter: Zoram Thanga >Assignee: Zoram Thanga >Priority: Minor > Labels: supportability > > I recently had to troubleshoot JDBC client connection slowness, and looking > through this part of the code suggests we could improve the logging to make > debugging this kind of issue easier. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7444) Improve debug logging of session opening and expiry
Zoram Thanga created IMPALA-7444: Summary: Improve debug logging of session opening and expiry Key: IMPALA-7444 URL: https://issues.apache.org/jira/browse/IMPALA-7444 Project: IMPALA Issue Type: Improvement Components: Backend Affects Versions: Impala 2.12.0, Impala 3.0, Impala 3.1.0 Reporter: Zoram Thanga Assignee: Zoram Thanga I recently had to troubleshoot JDBC client connection slowness, and looking through this part of the code suggests we could improve the logging to make debugging this kind of issue easier. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-7444) Improve debug logging of session opening and expiry
Zoram Thanga created IMPALA-7444: Summary: Improve debug logging of session opening and expiry Key: IMPALA-7444 URL: https://issues.apache.org/jira/browse/IMPALA-7444 Project: IMPALA Issue Type: Improvement Components: Backend Affects Versions: Impala 2.12.0, Impala 3.0, Impala 3.1.0 Reporter: Zoram Thanga Assignee: Zoram Thanga I recently had to troubleshoot JDBC client connection slowness, and looking through this part of the code suggests we could improve the logging to make debugging this kind of issue easier. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-6671) Metadata operations that modify a table blocks topic updates for other unrelated operations
[
https://issues.apache.org/jira/browse/IMPALA-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga updated IMPALA-6671:
-
Affects Version/s: Impala 2.10.0
Impala 2.11.0
> Metadata operations that modify a table blocks topic updates for other
> unrelated operations
> ---
>
> Key: IMPALA-6671
> URL: https://issues.apache.org/jira/browse/IMPALA-6671
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 2.10.0, Impala 2.11.0, Impala 2.12.0
>Reporter: Mostafa Mokhtar
>Priority: Critical
> Labels: catalog-server, perfomance
>
> Metadata operations that mutate the state of a table like "compute stats foo"
> or "alter recover partitions" block topic updates for read only operations
> against unrelated tables as "describe bar".
> Thread for blocked operation
> {code}
> "Thread-7" prio=10 tid=0x11613000 nid=0x21b3b waiting on condition
> [0x7f5f2ef52000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f6f57ff0240> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.addTableToCatalogDeltaHelper(CatalogServiceCatalog.java:639)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.addTableToCatalogDelta(CatalogServiceCatalog.java:611)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.addDatabaseToCatalogDelta(CatalogServiceCatalog.java:567)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.getCatalogDelta(CatalogServiceCatalog.java:449)
> at
> org.apache.impala.service.JniCatalog.getCatalogDelta(JniCatalog.java:126)
> {code}
> Thread for blocking operation
> {code}
> "Thread-130" prio=10 tid=0x113d5800 nid=0x2499d runnable
> [0x7f5ef80d]
>java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x7f5fffcd9f18> (a java.io.BufferedInputStream)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:346)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:423)
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:405)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_add_partitions_req(ThriftHiveMetastore.java:1639)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.add_partitions_req(ThriftHiveMetastore.java:1626)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.add_partitions(HiveMetaStoreClient.java:609)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at
[jira] [Commented] (IMPALA-6671) Metadata operations that modify a table blocks topic updates for other unrelated operations
[
https://issues.apache.org/jira/browse/IMPALA-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555934#comment-16555934
]
Zoram Thanga commented on IMPALA-6671:
--
This issue has existed since Impala 2.10 (or may be even earlier).
> Metadata operations that modify a table blocks topic updates for other
> unrelated operations
> ---
>
> Key: IMPALA-6671
> URL: https://issues.apache.org/jira/browse/IMPALA-6671
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 2.10.0, Impala 2.11.0, Impala 2.12.0
>Reporter: Mostafa Mokhtar
>Priority: Critical
> Labels: catalog-server, perfomance
>
> Metadata operations that mutate the state of a table like "compute stats foo"
> or "alter recover partitions" block topic updates for read only operations
> against unrelated tables as "describe bar".
> Thread for blocked operation
> {code}
> "Thread-7" prio=10 tid=0x11613000 nid=0x21b3b waiting on condition
> [0x7f5f2ef52000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f6f57ff0240> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.addTableToCatalogDeltaHelper(CatalogServiceCatalog.java:639)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.addTableToCatalogDelta(CatalogServiceCatalog.java:611)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.addDatabaseToCatalogDelta(CatalogServiceCatalog.java:567)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.getCatalogDelta(CatalogServiceCatalog.java:449)
> at
> org.apache.impala.service.JniCatalog.getCatalogDelta(JniCatalog.java:126)
> {code}
> Thread for blocking operation
> {code}
> "Thread-130" prio=10 tid=0x113d5800 nid=0x2499d runnable
> [0x7f5ef80d]
>java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x7f5fffcd9f18> (a java.io.BufferedInputStream)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:346)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:423)
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:405)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_add_partitions_req(ThriftHiveMetastore.java:1639)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.add_partitions_req(ThriftHiveMetastore.java:1626)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.add_partitions(HiveMetaStoreClient.java:609)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
[jira] [Commented] (IMPALA-5058) Improve concurrency of DDL/DML operations during catalog updates
[ https://issues.apache.org/jira/browse/IMPALA-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554524#comment-16554524 ] Zoram Thanga commented on IMPALA-5058: -- [~tge...@cloudera.com], this is fixed in CDH 5.15.0. > Improve concurrency of DDL/DML operations during catalog updates > > > Key: IMPALA-5058 > URL: https://issues.apache.org/jira/browse/IMPALA-5058 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Affects Versions: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0 >Reporter: Dimitris Tsirogiannis >Assignee: Dimitris Tsirogiannis >Priority: Critical > Labels: catalog-server, performance, usability > Fix For: Impala 2.12.0 > > Attachments: sample-refresh-duration-graph.png > > > Currently, long running DDL/DML operations can block other operations from > making progress if they run concurrently with the getCatalogObjects() call > that creates catalog updates. The reason is that while getCatalogObjects() > holds the lock for its entire duration and also tries to acquire the locks > for the tables it processes. If that operation is blocked by another > operation on a table then any other, unrelated, catalog write operation > cannot make any progress as it cannot acquire the catalog lock which is held > by getCatalogObjects(). > From a user's point of view, concurrent DDL/DML operations are executed > serially and, consequently, the latency of DDL/DML operations may vary > significantly. With the fix for this issue concurrent DDL/DML operations > should allow to run concurrently and the throughput of these operations > should increase significantly. At the same time, the latency of DDL/DML > operations should not depend on any other operations that are running at the > same time. It's important to note that when we talk about the latency of an > operation it is with respect to the coordinator that initiates the operation; > the fix doesn't do anything to improve the latency of broadcasting metadata > changes through the statestore. Some common user case where this fix is > applicable are the following: > # Concurrent REFRESH operations on different tables. > # Concurrent ALTER TABLE operations on different tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7014) Disable stacktrace symbolisation by default
[
https://issues.apache.org/jira/browse/IMPALA-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga resolved IMPALA-7014.
--
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Disable stacktrace symbolisation by default
> ---
>
> Key: IMPALA-7014
> URL: https://issues.apache.org/jira/browse/IMPALA-7014
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Not Applicable
>Reporter: Tim Armstrong
>Assignee: Zoram Thanga
>Priority: Critical
> Fix For: Impala 3.1.0
>
>
> We got burned by the code of producing stacktrace again with IMPALA-6996. I
> did a quick investigation into this, based on the hypothesis that the
> symbolisation was the expensive part, rather than getting the addresses. I
> added a stopwatch to GetStackTrace() to measure the time in nanoseconds and
> ran a test that produces a backtrace
> The first experiment was
> {noformat}
> $ start-impala-cluster.py --impalad_args='--symbolize_stacktrace=true' &&
> impala-py.test tests/query_test/test_scanners.py -k codec
> I0511 09:45:11.897944 30904 debug-util.cc:283] stacktrace time: 75175573
> I0511 09:45:11.897956 30904 status.cc:125] File
> 'hdfs://localhost:20500/test-warehouse/test_bad_compression_codec_308108.db/bad_codec/bad_codec.parquet'
> uses an unsupported compression: 5000 for column 'id'.
> @ 0x18782ef impala::Status::Status()
> @ 0x2cbe96f
> impala::ParquetMetadataUtils::ValidateRowGroupColumn()
> @ 0x205f597 impala::BaseScalarColumnReader::Reset()
> @ 0x1feebe6 impala::HdfsParquetScanner::InitScalarColumns()
> @ 0x1fe6ff3 impala::HdfsParquetScanner::NextRowGroup()
> @ 0x1fe58d8 impala::HdfsParquetScanner::GetNextInternal()
> @ 0x1fe3eea impala::HdfsParquetScanner::ProcessSplit()
> @ 0x1f6ba36 impala::HdfsScanNode::ProcessSplit()
> @ 0x1f6adc4 impala::HdfsScanNode::ScannerThread()
> @ 0x1f6a1c4
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x1f6c2a6
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x1bd3b1a boost::function0<>::operator()()
> @ 0x1ebecd5 impala::Thread::SuperviseThread()
> @ 0x1ec6e71 boost::_bi::list5<>::operator()<>()
> @ 0x1ec6d95 boost::_bi::bind_t<>::operator()()
> @ 0x1ec6d58 boost::detail::thread_data<>::run()
> @ 0x31b3ada thread_proxy
> @ 0x7f9be67d36ba start_thread
> @ 0x7f9be650941d clone
> {noformat}
> The stacktrace took 75ms, which is pretty bad! It would be worse on a
> production system with more memory maps.
> The next experiment was to disable it:
> {noformat}
> start-impala-cluster.py --impalad_args='--symbolize_stacktrace=false' &&
> impala-py.test tests/query_test/test_scanners.py -k codec
> I0511 09:43:47.574185 29514 debug-util.cc:283] stacktrace time: 29528
> I0511 09:43:47.574193 29514 status.cc:125] File
> 'hdfs://localhost:20500/test-warehouse/test_bad_compression_codec_cb5d0225.db/bad_codec/bad_codec.parquet'
> uses an unsupported compression: 5000 for column 'id'.
> @ 0x18782ef
> @ 0x2cbe96f
> @ 0x205f597
> @ 0x1feebe6
> @ 0x1fe6ff3
> @ 0x1fe58d8
> @ 0x1fe3eea
> @ 0x1f6ba36
> @ 0x1f6adc4
> @ 0x1f6a1c4
> @ 0x1f6c2a6
> @ 0x1bd3b1a
> @ 0x1ebecd5
> @ 0x1ec6e71
> @ 0x1ec6d95
> @ 0x1ec6d58
> @ 0x31b3ada
> @ 0x7fbdcbdef6ba
> @ 0x7fbdcbb2541d
> {noformat}
> That's 2545x faster! If the addresses are in the statically linked binary, we
> can use addrline to get back the line numbers:
> {noformat}
> $ addr2line -e be/build/latest/service/impalad 0x2cbe96f
> /home/tarmstrong/Impala/incubator-impala/be/src/exec/parquet-metadata-utils.cc:166
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7014) Disable stacktrace symbolisation by default
[
https://issues.apache.org/jira/browse/IMPALA-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga resolved IMPALA-7014.
--
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Disable stacktrace symbolisation by default
> ---
>
> Key: IMPALA-7014
> URL: https://issues.apache.org/jira/browse/IMPALA-7014
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Not Applicable
>Reporter: Tim Armstrong
>Assignee: Zoram Thanga
>Priority: Critical
> Fix For: Impala 3.1.0
>
>
> We got burned by the code of producing stacktrace again with IMPALA-6996. I
> did a quick investigation into this, based on the hypothesis that the
> symbolisation was the expensive part, rather than getting the addresses. I
> added a stopwatch to GetStackTrace() to measure the time in nanoseconds and
> ran a test that produces a backtrace
> The first experiment was
> {noformat}
> $ start-impala-cluster.py --impalad_args='--symbolize_stacktrace=true' &&
> impala-py.test tests/query_test/test_scanners.py -k codec
> I0511 09:45:11.897944 30904 debug-util.cc:283] stacktrace time: 75175573
> I0511 09:45:11.897956 30904 status.cc:125] File
> 'hdfs://localhost:20500/test-warehouse/test_bad_compression_codec_308108.db/bad_codec/bad_codec.parquet'
> uses an unsupported compression: 5000 for column 'id'.
> @ 0x18782ef impala::Status::Status()
> @ 0x2cbe96f
> impala::ParquetMetadataUtils::ValidateRowGroupColumn()
> @ 0x205f597 impala::BaseScalarColumnReader::Reset()
> @ 0x1feebe6 impala::HdfsParquetScanner::InitScalarColumns()
> @ 0x1fe6ff3 impala::HdfsParquetScanner::NextRowGroup()
> @ 0x1fe58d8 impala::HdfsParquetScanner::GetNextInternal()
> @ 0x1fe3eea impala::HdfsParquetScanner::ProcessSplit()
> @ 0x1f6ba36 impala::HdfsScanNode::ProcessSplit()
> @ 0x1f6adc4 impala::HdfsScanNode::ScannerThread()
> @ 0x1f6a1c4
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x1f6c2a6
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x1bd3b1a boost::function0<>::operator()()
> @ 0x1ebecd5 impala::Thread::SuperviseThread()
> @ 0x1ec6e71 boost::_bi::list5<>::operator()<>()
> @ 0x1ec6d95 boost::_bi::bind_t<>::operator()()
> @ 0x1ec6d58 boost::detail::thread_data<>::run()
> @ 0x31b3ada thread_proxy
> @ 0x7f9be67d36ba start_thread
> @ 0x7f9be650941d clone
> {noformat}
> The stacktrace took 75ms, which is pretty bad! It would be worse on a
> production system with more memory maps.
> The next experiment was to disable it:
> {noformat}
> start-impala-cluster.py --impalad_args='--symbolize_stacktrace=false' &&
> impala-py.test tests/query_test/test_scanners.py -k codec
> I0511 09:43:47.574185 29514 debug-util.cc:283] stacktrace time: 29528
> I0511 09:43:47.574193 29514 status.cc:125] File
> 'hdfs://localhost:20500/test-warehouse/test_bad_compression_codec_cb5d0225.db/bad_codec/bad_codec.parquet'
> uses an unsupported compression: 5000 for column 'id'.
> @ 0x18782ef
> @ 0x2cbe96f
> @ 0x205f597
> @ 0x1feebe6
> @ 0x1fe6ff3
> @ 0x1fe58d8
> @ 0x1fe3eea
> @ 0x1f6ba36
> @ 0x1f6adc4
> @ 0x1f6a1c4
> @ 0x1f6c2a6
> @ 0x1bd3b1a
> @ 0x1ebecd5
> @ 0x1ec6e71
> @ 0x1ec6d95
> @ 0x1ec6d58
> @ 0x31b3ada
> @ 0x7fbdcbdef6ba
> @ 0x7fbdcbb2541d
> {noformat}
> That's 2545x faster! If the addresses are in the statically linked binary, we
> can use addrline to get back the line numbers:
> {noformat}
> $ addr2line -e be/build/latest/service/impalad 0x2cbe96f
> /home/tarmstrong/Impala/incubator-impala/be/src/exec/parquet-metadata-utils.cc:166
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-6086) Use of permanent function should require SELECT privilege on DB
[ https://issues.apache.org/jira/browse/IMPALA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga resolved IMPALA-6086. -- Resolution: Fixed Fix Version/s: Impala 3.1.0 > Use of permanent function should require SELECT privilege on DB > --- > > Key: IMPALA-6086 > URL: https://issues.apache.org/jira/browse/IMPALA-6086 > Project: IMPALA > Issue Type: Bug > Components: Catalog, Security >Affects Versions: Impala 2.9.0, Impala 3.1.0 >Reporter: Zoram Thanga >Assignee: Zoram Thanga >Priority: Minor > Labels: security > Fix For: Impala 3.1.0 > > > A user that has no privilege on a database should not be able to execute any > permanent functions in that database. This is currently possible, and should > be fixed, so that the user must have SELECT privilege to execute permanent > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-6086) Use of permanent function should require SELECT privilege on DB
[ https://issues.apache.org/jira/browse/IMPALA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga resolved IMPALA-6086. -- Resolution: Fixed Fix Version/s: Impala 3.1.0 > Use of permanent function should require SELECT privilege on DB > --- > > Key: IMPALA-6086 > URL: https://issues.apache.org/jira/browse/IMPALA-6086 > Project: IMPALA > Issue Type: Bug > Components: Catalog, Security >Affects Versions: Impala 2.9.0, Impala 3.1.0 >Reporter: Zoram Thanga >Assignee: Zoram Thanga >Priority: Minor > Labels: security > Fix For: Impala 3.1.0 > > > A user that has no privilege on a database should not be able to execute any > permanent functions in that database. This is currently possible, and should > be fixed, so that the user must have SELECT privilege to execute permanent > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IMPALA-7288) Codegen crash in FinalizeModule()
[
https://issues.apache.org/jira/browse/IMPALA-7288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542024#comment-16542024
]
Zoram Thanga commented on IMPALA-7288:
--
Possibly introduced by IMPALA-6177?
> Codegen crash in FinalizeModule()
> -
>
> Key: IMPALA-7288
> URL: https://issues.apache.org/jira/browse/IMPALA-7288
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.12.0, Impala 3.1.0
>Reporter: Balazs Jeszenszky
>Priority: Blocker
>
> The following sequence crashes Impala 2.12 reliably:
> {code}
> CREATE TABLE test (c1 CHAR(6),c2 CHAR(6));
> select 1 from test t1, test t2
> where t1.c1 = FROM_TIMESTAMP(cast(t2.c2 as string), 'MMdd');
> {code}
> hs_err_pid has:
> {code}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x03b36ce4, pid=28459, tid=0x7f2c49685700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_162-b12) (build
> 1.8.0_162-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # C [impalad+0x3736ce4] llvm::Value::getContext() const+0x4
> {code}
> Backtrace is:
> {code}
> #0 0x7f2cb217a5f7 in raise () from /lib64/libc.so.6
> #1 0x7f2cb217bce8 in abort () from /lib64/libc.so.6
> #2 0x7f2cb4de2f35 in os::abort(bool) () from
> /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #3 0x7f2cb4f86f33 in VMError::report_and_die() () from
> /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #4 0x7f2cb4de922f in JVM_handle_linux_signal () from
> /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #5 0x7f2cb4ddf253 in signalHandler(int, siginfo*, void*) () from
> /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x03b36ce4 in llvm::Value::getContext() const ()
> #8 0x03b36cff in llvm::Value::getValueName() const ()
> #9 0x03b36de9 in llvm::Value::getName() const ()
> #10 0x01ba6bb2 in impala::LlvmCodeGen::FinalizeModule (this=0x9b53980)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/be/src/codegen/llvm-codegen.cc:1076
> #11 0x018f5c0f in impala::FragmentInstanceState::Open (this=0xac0b400)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/be/src/runtime/fragment-instance-state.cc:255
> #12 0x018f3699 in impala::FragmentInstanceState::Exec (this=0xac0b400)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/be/src/runtime/fragment-instance-state.cc:80
> #13 0x019028c3 in impala::QueryState::ExecFInstance (this=0x9c6ad00,
> fis=0xac0b400)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/be/src/runtime/query-state.cc:410
> #14 0x0190113c in impala::QueryStateoperator()(void)
> const (__closure=0x7f2c49684be8)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/be/src/runtime/query-state.cc:350
> #15 0x019034dd in
> boost::detail::function::void_function_obj_invoker0,
> void>::invoke(boost::detail::function::function_buffer &)
> (function_obj_ptr=...)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> {code}
> Crash is at
> https://github.com/cloudera/Impala/blob/cdh5-2.12.0_5.15.0/be/src/codegen/llvm-codegen.cc#L1070-L1079.
> The repro steps seem to be quite specific.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7288) Codegen crash in FinalizeModule()
[
https://issues.apache.org/jira/browse/IMPALA-7288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga updated IMPALA-7288:
-
Affects Version/s: Impala 3.1.0
> Codegen crash in FinalizeModule()
> -
>
> Key: IMPALA-7288
> URL: https://issues.apache.org/jira/browse/IMPALA-7288
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.12.0, Impala 3.1.0
>Reporter: Balazs Jeszenszky
>Priority: Blocker
>
> The following sequence crashes Impala 2.12 reliably:
> {code}
> CREATE TABLE test (c1 CHAR(6),c2 CHAR(6));
> select 1 from test t1, test t2
> where t1.c1 = FROM_TIMESTAMP(cast(t2.c2 as string), 'MMdd');
> {code}
> hs_err_pid has:
> {code}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x03b36ce4, pid=28459, tid=0x7f2c49685700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_162-b12) (build
> 1.8.0_162-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # C [impalad+0x3736ce4] llvm::Value::getContext() const+0x4
> {code}
> Backtrace is:
> {code}
> #0 0x7f2cb217a5f7 in raise () from /lib64/libc.so.6
> #1 0x7f2cb217bce8 in abort () from /lib64/libc.so.6
> #2 0x7f2cb4de2f35 in os::abort(bool) () from
> /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #3 0x7f2cb4f86f33 in VMError::report_and_die() () from
> /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #4 0x7f2cb4de922f in JVM_handle_linux_signal () from
> /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #5 0x7f2cb4ddf253 in signalHandler(int, siginfo*, void*) () from
> /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x03b36ce4 in llvm::Value::getContext() const ()
> #8 0x03b36cff in llvm::Value::getValueName() const ()
> #9 0x03b36de9 in llvm::Value::getName() const ()
> #10 0x01ba6bb2 in impala::LlvmCodeGen::FinalizeModule (this=0x9b53980)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/be/src/codegen/llvm-codegen.cc:1076
> #11 0x018f5c0f in impala::FragmentInstanceState::Open (this=0xac0b400)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/be/src/runtime/fragment-instance-state.cc:255
> #12 0x018f3699 in impala::FragmentInstanceState::Exec (this=0xac0b400)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/be/src/runtime/fragment-instance-state.cc:80
> #13 0x019028c3 in impala::QueryState::ExecFInstance (this=0x9c6ad00,
> fis=0xac0b400)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/be/src/runtime/query-state.cc:410
> #14 0x0190113c in impala::QueryStateoperator()(void)
> const (__closure=0x7f2c49684be8)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/be/src/runtime/query-state.cc:350
> #15 0x019034dd in
> boost::detail::function::void_function_obj_invoker0,
> void>::invoke(boost::detail::function::function_buffer &)
> (function_obj_ptr=...)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> {code}
> Crash is at
> https://github.com/cloudera/Impala/blob/cdh5-2.12.0_5.15.0/be/src/codegen/llvm-codegen.cc#L1070-L1079.
> The repro steps seem to be quite specific.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7288) Codegen crash in FinalizeModule()
[
https://issues.apache.org/jira/browse/IMPALA-7288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542004#comment-16542004
]
Zoram Thanga commented on IMPALA-7288:
--
FYI, the crash repro's with the latest upstream code as well, so LLVM 3.9 or
5.1 does not seem to matter. Stating the obvious, we don't crash if
DISABLE_CODEGEN=TRUE.
> Codegen crash in FinalizeModule()
> -
>
> Key: IMPALA-7288
> URL: https://issues.apache.org/jira/browse/IMPALA-7288
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.12.0, Impala 3.1.0
>Reporter: Balazs Jeszenszky
>Priority: Blocker
>
> The following sequence crashes Impala 2.12 reliably:
> {code}
> CREATE TABLE test (c1 CHAR(6),c2 CHAR(6));
> select 1 from test t1, test t2
> where t1.c1 = FROM_TIMESTAMP(cast(t2.c2 as string), 'MMdd');
> {code}
> hs_err_pid has:
> {code}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x03b36ce4, pid=28459, tid=0x7f2c49685700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_162-b12) (build
> 1.8.0_162-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # C [impalad+0x3736ce4] llvm::Value::getContext() const+0x4
> {code}
> Backtrace is:
> {code}
> #0 0x7f2cb217a5f7 in raise () from /lib64/libc.so.6
> #1 0x7f2cb217bce8 in abort () from /lib64/libc.so.6
> #2 0x7f2cb4de2f35 in os::abort(bool) () from
> /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #3 0x7f2cb4f86f33 in VMError::report_and_die() () from
> /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #4 0x7f2cb4de922f in JVM_handle_linux_signal () from
> /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #5 0x7f2cb4ddf253 in signalHandler(int, siginfo*, void*) () from
> /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x03b36ce4 in llvm::Value::getContext() const ()
> #8 0x03b36cff in llvm::Value::getValueName() const ()
> #9 0x03b36de9 in llvm::Value::getName() const ()
> #10 0x01ba6bb2 in impala::LlvmCodeGen::FinalizeModule (this=0x9b53980)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/be/src/codegen/llvm-codegen.cc:1076
> #11 0x018f5c0f in impala::FragmentInstanceState::Open (this=0xac0b400)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/be/src/runtime/fragment-instance-state.cc:255
> #12 0x018f3699 in impala::FragmentInstanceState::Exec (this=0xac0b400)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/be/src/runtime/fragment-instance-state.cc:80
> #13 0x019028c3 in impala::QueryState::ExecFInstance (this=0x9c6ad00,
> fis=0xac0b400)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/be/src/runtime/query-state.cc:410
> #14 0x0190113c in impala::QueryStateoperator()(void)
> const (__closure=0x7f2c49684be8)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/be/src/runtime/query-state.cc:350
> #15 0x019034dd in
> boost::detail::function::void_function_obj_invoker0,
> void>::invoke(boost::detail::function::function_buffer &)
> (function_obj_ptr=...)
> at
> /usr/src/debug/impala-2.12.0-cdh5.15.0/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> {code}
> Crash is at
> https://github.com/cloudera/Impala/blob/cdh5-2.12.0_5.15.0/be/src/codegen/llvm-codegen.cc#L1070-L1079.
> The repro steps seem to be quite specific.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7285) Restructure analyzer code to perform security/access checks before re-analysis
Zoram Thanga created IMPALA-7285:
Summary: Restructure analyzer code to perform security/access
checks before re-analysis
Key: IMPALA-7285
URL: https://issues.apache.org/jira/browse/IMPALA-7285
Project: IMPALA
Issue Type: Bug
Components: Frontend, Security
Affects Versions: Impala 2.12.0
Reporter: Zoram Thanga
This came out during the review for
[https://gerrit.cloudera.org/#/c/10850/|https://gerrit.cloudera.org/#/c/10850/]:
{quote}
supposing the folded fn reveals something interesting, e.g., getSSN("some user
name") ... this approach evaluates it and outputs it to the log. while I don't
think we output this rewritten query in an error (or possibly elsewhere
downstream), have you looked at avoiding the evaluation of fn in the first
place if access is not permitted? the approach here seems prone to currently
leak and can get worse depending on future changes.
{quote}
In general, it would be good to handle query analysis in the following sequence:
# Parse
# Check security/access
# Analyze
# Re-write
# Re-analyze
# etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7285) Restructure analyzer code to perform security/access checks before re-analysis
Zoram Thanga created IMPALA-7285:
Summary: Restructure analyzer code to perform security/access
checks before re-analysis
Key: IMPALA-7285
URL: https://issues.apache.org/jira/browse/IMPALA-7285
Project: IMPALA
Issue Type: Bug
Components: Frontend, Security
Affects Versions: Impala 2.12.0
Reporter: Zoram Thanga
This came out during the review for
[https://gerrit.cloudera.org/#/c/10850/|https://gerrit.cloudera.org/#/c/10850/]:
{quote}
supposing the folded fn reveals something interesting, e.g., getSSN("some user
name") ... this approach evaluates it and outputs it to the log. while I don't
think we output this rewritten query in an error (or possibly elsewhere
downstream), have you looked at avoiding the evaluation of fn in the first
place if access is not permitted? the approach here seems prone to currently
leak and can get worse depending on future changes.
{quote}
In general, it would be good to handle query analysis in the following sequence:
# Parse
# Check security/access
# Analyze
# Re-write
# Re-analyze
# etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (IMPALA-7178) Reduce logging for common data errors
[ https://issues.apache.org/jira/browse/IMPALA-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16537325#comment-16537325 ] Zoram Thanga edited comment on IMPALA-7178 at 7/9/18 5:47 PM: -- I wonder if IMPALA-6971 is an instance of this log flooding? was (Author: zoram): I wonder if IMPALA-6996 is an instance of this log flooding? > Reduce logging for common data errors > - > > Key: IMPALA-7178 > URL: https://issues.apache.org/jira/browse/IMPALA-7178 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Csaba Ringhofer >Assignee: Csaba Ringhofer >Priority: Major > > Some data errors (for example out-of-range parquet timestamps) can dominate > logs if a table contains a large number of rows with invalid data. If an > error has its own error code (see common/thrift/generate_error_codes.py), > then these errors are already aggregated to the user > (RuntimeState::LogError()) for every query, but the logs will contain a new > line for every occurrence. This is not too useful most of times, as the log > lines will repeat the same information (the corrupt data itself is not logged > as it can be sensitive information). > The best would to reduce logging without loosing information: > - the first occurrence of an error should be logged (per > query/fragment/table/file/column) to help the investigation of cases where > the data error leads to other errors and to avoid breaking log analyzer tools > that search for the current format > - other occurrences can be aggregated, like "in query Q table T column C XY > error occurred N times" > An extra goal is to avoid calling RuntimeState::LogError() for other > occurrences than the first one, as RuntimeState::LogError() uses a (per > fragment) lock. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7178) Reduce logging for common data errors
[ https://issues.apache.org/jira/browse/IMPALA-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16537325#comment-16537325 ] Zoram Thanga commented on IMPALA-7178: -- I wonder if IMPALA-6996 is an instance of this log flooding? > Reduce logging for common data errors > - > > Key: IMPALA-7178 > URL: https://issues.apache.org/jira/browse/IMPALA-7178 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Csaba Ringhofer >Assignee: Csaba Ringhofer >Priority: Major > > Some data errors (for example out-of-range parquet timestamps) can dominate > logs if a table contains a large number of rows with invalid data. If an > error has its own error code (see common/thrift/generate_error_codes.py), > then these errors are already aggregated to the user > (RuntimeState::LogError()) for every query, but the logs will contain a new > line for every occurrence. This is not too useful most of times, as the log > lines will repeat the same information (the corrupt data itself is not logged > as it can be sensitive information). > The best would to reduce logging without loosing information: > - the first occurrence of an error should be logged (per > query/fragment/table/file/column) to help the investigation of cases where > the data error leads to other errors and to avoid breaking log analyzer tools > that search for the current format > - other occurrences can be aggregated, like "in query Q table T column C XY > error occurred N times" > An extra goal is to avoid calling RuntimeState::LogError() for other > occurrences than the first one, as RuntimeState::LogError() uses a (per > fragment) lock. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7014) Disable stacktrace symbolisation by default
[
https://issues.apache.org/jira/browse/IMPALA-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga reassigned IMPALA-7014:
Assignee: Zoram Thanga (was: Joe McDonnell)
> Disable stacktrace symbolisation by default
> ---
>
> Key: IMPALA-7014
> URL: https://issues.apache.org/jira/browse/IMPALA-7014
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Not Applicable
>Reporter: Tim Armstrong
>Assignee: Zoram Thanga
>Priority: Critical
>
> We got burned by the code of producing stacktrace again with IMPALA-6996. I
> did a quick investigation into this, based on the hypothesis that the
> symbolisation was the expensive part, rather than getting the addresses. I
> added a stopwatch to GetStackTrace() to measure the time in nanoseconds and
> ran a test that produces a backtrace
> The first experiment was
> {noformat}
> $ start-impala-cluster.py --impalad_args='--symbolize_stacktrace=true' &&
> impala-py.test tests/query_test/test_scanners.py -k codec
> I0511 09:45:11.897944 30904 debug-util.cc:283] stacktrace time: 75175573
> I0511 09:45:11.897956 30904 status.cc:125] File
> 'hdfs://localhost:20500/test-warehouse/test_bad_compression_codec_308108.db/bad_codec/bad_codec.parquet'
> uses an unsupported compression: 5000 for column 'id'.
> @ 0x18782ef impala::Status::Status()
> @ 0x2cbe96f
> impala::ParquetMetadataUtils::ValidateRowGroupColumn()
> @ 0x205f597 impala::BaseScalarColumnReader::Reset()
> @ 0x1feebe6 impala::HdfsParquetScanner::InitScalarColumns()
> @ 0x1fe6ff3 impala::HdfsParquetScanner::NextRowGroup()
> @ 0x1fe58d8 impala::HdfsParquetScanner::GetNextInternal()
> @ 0x1fe3eea impala::HdfsParquetScanner::ProcessSplit()
> @ 0x1f6ba36 impala::HdfsScanNode::ProcessSplit()
> @ 0x1f6adc4 impala::HdfsScanNode::ScannerThread()
> @ 0x1f6a1c4
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x1f6c2a6
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x1bd3b1a boost::function0<>::operator()()
> @ 0x1ebecd5 impala::Thread::SuperviseThread()
> @ 0x1ec6e71 boost::_bi::list5<>::operator()<>()
> @ 0x1ec6d95 boost::_bi::bind_t<>::operator()()
> @ 0x1ec6d58 boost::detail::thread_data<>::run()
> @ 0x31b3ada thread_proxy
> @ 0x7f9be67d36ba start_thread
> @ 0x7f9be650941d clone
> {noformat}
> The stacktrace took 75ms, which is pretty bad! It would be worse on a
> production system with more memory maps.
> The next experiment was to disable it:
> {noformat}
> start-impala-cluster.py --impalad_args='--symbolize_stacktrace=false' &&
> impala-py.test tests/query_test/test_scanners.py -k codec
> I0511 09:43:47.574185 29514 debug-util.cc:283] stacktrace time: 29528
> I0511 09:43:47.574193 29514 status.cc:125] File
> 'hdfs://localhost:20500/test-warehouse/test_bad_compression_codec_cb5d0225.db/bad_codec/bad_codec.parquet'
> uses an unsupported compression: 5000 for column 'id'.
> @ 0x18782ef
> @ 0x2cbe96f
> @ 0x205f597
> @ 0x1feebe6
> @ 0x1fe6ff3
> @ 0x1fe58d8
> @ 0x1fe3eea
> @ 0x1f6ba36
> @ 0x1f6adc4
> @ 0x1f6a1c4
> @ 0x1f6c2a6
> @ 0x1bd3b1a
> @ 0x1ebecd5
> @ 0x1ec6e71
> @ 0x1ec6d95
> @ 0x1ec6d58
> @ 0x31b3ada
> @ 0x7fbdcbdef6ba
> @ 0x7fbdcbb2541d
> {noformat}
> That's 2545x faster! If the addresses are in the statically linked binary, we
> can use addrline to get back the line numbers:
> {noformat}
> $ addr2line -e be/build/latest/service/impalad 0x2cbe96f
> /home/tarmstrong/Impala/incubator-impala/be/src/exec/parquet-metadata-utils.cc:166
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7014) Disable stacktrace symbolisation by default
[
https://issues.apache.org/jira/browse/IMPALA-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-7014 started by Zoram Thanga.
> Disable stacktrace symbolisation by default
> ---
>
> Key: IMPALA-7014
> URL: https://issues.apache.org/jira/browse/IMPALA-7014
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Not Applicable
>Reporter: Tim Armstrong
>Assignee: Zoram Thanga
>Priority: Critical
>
> We got burned by the code of producing stacktrace again with IMPALA-6996. I
> did a quick investigation into this, based on the hypothesis that the
> symbolisation was the expensive part, rather than getting the addresses. I
> added a stopwatch to GetStackTrace() to measure the time in nanoseconds and
> ran a test that produces a backtrace
> The first experiment was
> {noformat}
> $ start-impala-cluster.py --impalad_args='--symbolize_stacktrace=true' &&
> impala-py.test tests/query_test/test_scanners.py -k codec
> I0511 09:45:11.897944 30904 debug-util.cc:283] stacktrace time: 75175573
> I0511 09:45:11.897956 30904 status.cc:125] File
> 'hdfs://localhost:20500/test-warehouse/test_bad_compression_codec_308108.db/bad_codec/bad_codec.parquet'
> uses an unsupported compression: 5000 for column 'id'.
> @ 0x18782ef impala::Status::Status()
> @ 0x2cbe96f
> impala::ParquetMetadataUtils::ValidateRowGroupColumn()
> @ 0x205f597 impala::BaseScalarColumnReader::Reset()
> @ 0x1feebe6 impala::HdfsParquetScanner::InitScalarColumns()
> @ 0x1fe6ff3 impala::HdfsParquetScanner::NextRowGroup()
> @ 0x1fe58d8 impala::HdfsParquetScanner::GetNextInternal()
> @ 0x1fe3eea impala::HdfsParquetScanner::ProcessSplit()
> @ 0x1f6ba36 impala::HdfsScanNode::ProcessSplit()
> @ 0x1f6adc4 impala::HdfsScanNode::ScannerThread()
> @ 0x1f6a1c4
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x1f6c2a6
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x1bd3b1a boost::function0<>::operator()()
> @ 0x1ebecd5 impala::Thread::SuperviseThread()
> @ 0x1ec6e71 boost::_bi::list5<>::operator()<>()
> @ 0x1ec6d95 boost::_bi::bind_t<>::operator()()
> @ 0x1ec6d58 boost::detail::thread_data<>::run()
> @ 0x31b3ada thread_proxy
> @ 0x7f9be67d36ba start_thread
> @ 0x7f9be650941d clone
> {noformat}
> The stacktrace took 75ms, which is pretty bad! It would be worse on a
> production system with more memory maps.
> The next experiment was to disable it:
> {noformat}
> start-impala-cluster.py --impalad_args='--symbolize_stacktrace=false' &&
> impala-py.test tests/query_test/test_scanners.py -k codec
> I0511 09:43:47.574185 29514 debug-util.cc:283] stacktrace time: 29528
> I0511 09:43:47.574193 29514 status.cc:125] File
> 'hdfs://localhost:20500/test-warehouse/test_bad_compression_codec_cb5d0225.db/bad_codec/bad_codec.parquet'
> uses an unsupported compression: 5000 for column 'id'.
> @ 0x18782ef
> @ 0x2cbe96f
> @ 0x205f597
> @ 0x1feebe6
> @ 0x1fe6ff3
> @ 0x1fe58d8
> @ 0x1fe3eea
> @ 0x1f6ba36
> @ 0x1f6adc4
> @ 0x1f6a1c4
> @ 0x1f6c2a6
> @ 0x1bd3b1a
> @ 0x1ebecd5
> @ 0x1ec6e71
> @ 0x1ec6d95
> @ 0x1ec6d58
> @ 0x31b3ada
> @ 0x7fbdcbdef6ba
> @ 0x7fbdcbb2541d
> {noformat}
> That's 2545x faster! If the addresses are in the statically linked binary, we
> can use addrline to get back the line numbers:
> {noformat}
> $ addr2line -e be/build/latest/service/impalad 0x2cbe96f
> /home/tarmstrong/Impala/incubator-impala/be/src/exec/parquet-metadata-utils.cc:166
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7194) Impala crashed at FragmentInstanceState::Open()
[
https://issues.apache.org/jira/browse/IMPALA-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531685#comment-16531685
]
Zoram Thanga commented on IMPALA-7194:
--
Lowering priority to Critical, since this issue has been seen only once, and is
not reproducible so far.
> Impala crashed at FragmentInstanceState::Open()
> ---
>
> Key: IMPALA-7194
> URL: https://issues.apache.org/jira/browse/IMPALA-7194
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Xiaomin Zhang
>Priority: Critical
> Labels: crash
>
> Below is the crash stack resolved from minidump
> {code}
> Thread 3473 (crashed)
> 0 0x7fb0222e9000
> 1 impalad!impala::FragmentInstanceState::Open() [fragment-instance-state.cc :
> 255 + 0x11]
> 2 impalad!impala::FragmentInstanceState::Exec() [fragment-instance-state.cc :
> 80 + 0xb]
> 3 impalad!impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> [query-state.cc : 382 + 0x10]
> 4 impalad!impala::Thread::SuperviseThread(std::string const&, std::string
> const&, boost::function, impala::Promise*)
> [function_template.hpp : 767 + 0x7]
> 5 impalad!boost::detail::thread_data (std::string const&, std::string const&, boost::function,
> impala::Promise), boost::_bi::list4,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value> > > >::run() [bind.hpp : 457 + 0x6]
> 6 impalad!thread_proxy + 0xda
> {code}
> We do not have further details about this crash, and we only saw it happened
> once.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7194) Impala crashed at FragmentInstanceState::Open()
[
https://issues.apache.org/jira/browse/IMPALA-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga updated IMPALA-7194:
-
Priority: Critical (was: Blocker)
> Impala crashed at FragmentInstanceState::Open()
> ---
>
> Key: IMPALA-7194
> URL: https://issues.apache.org/jira/browse/IMPALA-7194
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Xiaomin Zhang
>Priority: Critical
> Labels: crash
>
> Below is the crash stack resolved from minidump
> {code}
> Thread 3473 (crashed)
> 0 0x7fb0222e9000
> 1 impalad!impala::FragmentInstanceState::Open() [fragment-instance-state.cc :
> 255 + 0x11]
> 2 impalad!impala::FragmentInstanceState::Exec() [fragment-instance-state.cc :
> 80 + 0xb]
> 3 impalad!impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> [query-state.cc : 382 + 0x10]
> 4 impalad!impala::Thread::SuperviseThread(std::string const&, std::string
> const&, boost::function, impala::Promise*)
> [function_template.hpp : 767 + 0x7]
> 5 impalad!boost::detail::thread_data (std::string const&, std::string const&, boost::function,
> impala::Promise), boost::_bi::list4,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value> > > >::run() [bind.hpp : 457 + 0x6]
> 6 impalad!thread_proxy + 0xda
> {code}
> We do not have further details about this crash, and we only saw it happened
> once.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6086) Use of permanent function should require SELECT privilege on DB
[
https://issues.apache.org/jira/browse/IMPALA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514221#comment-16514221
]
Zoram Thanga commented on IMPALA-6086:
--
In case it was not clear, here's a sample session. User 'dummy_user' has no
privileges:
{noformat}
[localhost:21000] default> show roles;
Query: show roles
ERROR: AuthorizationException: User 'dummy_user' does not have privileges to
access the requested policy metadata or Sentry Service is unavailable.
[localhost:21000] default> show tables;
Query: show tables
ERROR: AuthorizationException: User 'dummy_user' does not have privileges to
access: default.*.*
[localhost:21000] default> select has_vowels('abcdef');
Query: select has_vowels('abcdef')
Query submitted at: 2018-06-15 11:22:39 (Coordinator:
http://zoram-desktop:25000)
Query progress can be monitored at:
http://zoram-desktop:25000/query_plan?query_id=8347149d8e057744:12f35393
+--+
| default.has_vowels('abcdef') |
+--+
| true |
+--+
Fetched 1 row(s) in 0.12s
{noformat}
With my patch above, the result is different:
{noformat}
[localhost:21000] default> show roles;
Query: show roles
ERROR: AuthorizationException: User 'dummy_user' does not have privileges to
access the requested policy metadata or Sentry Service is unavailable.
[localhost:21000] default> show tables;
Query: show tables
ERROR: AuthorizationException: User 'dummy_user' does not have privileges to
access: default.*.*
[localhost:21000] default> select has_vowels('abcdef');
Query: select has_vowels('abcdef')
Query submitted at: 2018-06-15 11:30:40 (Coordinator:
http://zoram-desktop:25000)
ERROR: AuthorizationException: User 'dummy_user' does not have privileges to
access: default
[localhost:21000] default>
{noformat}
What do you think, [~fredyw]?
> Use of permanent function should require SELECT privilege on DB
> ---
>
> Key: IMPALA-6086
> URL: https://issues.apache.org/jira/browse/IMPALA-6086
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Security
>Affects Versions: Impala 2.9.0, Impala 3.1.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Minor
>
> A user that has no privilege on a database should not be able to execute any
> permanent functions in that database. This is currently possible, and should
> be fixed, so that the user must have SELECT privilege to execute permanent
> functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6086) Use of permanent function should require SELECT privilege on DB
[
https://issues.apache.org/jira/browse/IMPALA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513154#comment-16513154
]
Zoram Thanga commented on IMPALA-6086:
--
Here's a rough diff for what I think should be done:
{code:java}
diff --git a/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
b/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
index 3e7f0cc..6256fdb 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
@@ -52,6 +52,7 @@ import com.google.common.base.Strings;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
+import com.google.common.collect.ImmutableList;
/**
* Wrapper class for parsing, analyzing and rewriting a SQL stmt.
*/
@@ -454,6 +455,9 @@ public class AnalysisContext {
List origColLabels =
Lists.newArrayList(analysisResult_.stmt_.getColLabels());
+ // Save old privileges
+ ImmutableList oldPrivReqs =
analysisResult_.analyzer_.getPrivilegeReqs();
+
// Re-analyze the stmt with a new analyzer.
analysisResult_.analyzer_ = createAnalyzer(stmtTableCache);
analysisResult_.stmt_.reset();
@@ -465,6 +469,11 @@ public class AnalysisContext {
if (LOG.isTraceEnabled()) {
LOG.trace("rewrittenStmt: " + analysisResult_.stmt_.toSql());
}
+
+ // Restore privilege requests found during the first pass
+ for (PrivilegeRequest req : oldPrivReqs) {
+analysisResult_.analyzer_.registerPrivReq(req);
+ }
if (isExplain) analysisResult_.stmt_.setIsExplain();
Preconditions.checkState(!analysisResult_.requiresSubqueryRewrite());
}
{code}
> Use of permanent function should require SELECT privilege on DB
> ---
>
> Key: IMPALA-6086
> URL: https://issues.apache.org/jira/browse/IMPALA-6086
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Security
>Affects Versions: Impala 2.9.0, Impala 3.1.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Minor
>
> A user that has no privilege on a database should not be able to execute any
> permanent functions in that database. This is currently possible, and should
> be fixed, so that the user must have SELECT privilege to execute permanent
> functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6086) Use of permanent function should require SELECT privilege on DB
[ https://issues.apache.org/jira/browse/IMPALA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513138#comment-16513138 ] Zoram Thanga commented on IMPALA-6086: -- Thanks for the inputs, [~tarmstrong], [~fredyw]. Agree trim() was probably a bad example. However, I have verified that the conclusion is still valid for a non-system UDF, for instance by creating the has_vowels() UDF example, as detailed [here|https://www.cloudera.com/documentation/enterprise/5-14-x/topics/impala_udf.html#udf_tutorial]. > Use of permanent function should require SELECT privilege on DB > --- > > Key: IMPALA-6086 > URL: https://issues.apache.org/jira/browse/IMPALA-6086 > Project: IMPALA > Issue Type: Bug > Components: Catalog, Security >Affects Versions: Impala 2.9.0, Impala 3.1.0 >Reporter: Zoram Thanga >Assignee: Zoram Thanga >Priority: Minor > > A user that has no privilege on a database should not be able to execute any > permanent functions in that database. This is currently possible, and should > be fixed, so that the user must have SELECT privilege to execute permanent > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-6664) Tag log statements with query-ids
[ https://issues.apache.org/jira/browse/IMPALA-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-6664 started by Zoram Thanga. > Tag log statements with query-ids > - > > Key: IMPALA-6664 > URL: https://issues.apache.org/jira/browse/IMPALA-6664 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Affects Versions: Impala 2.12.0 >Reporter: bharath v >Assignee: Zoram Thanga >Priority: Major > Labels: supportability > > The idea is to include query-id in each logged statement in > coordinator/finsts/planner. This helps to collect query specific logs by just > grep'ing through the logs with the query-id. > This is hopefully easier to implement, now that we have the query debug info > available in the thread context [1]. > [1] https://issues.apache.org/jira/browse/IMPALA-3703 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6665) Tag CatalogOp logs with query IDs
[ https://issues.apache.org/jira/browse/IMPALA-6665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga reassigned IMPALA-6665: Assignee: Zoram Thanga > Tag CatalogOp logs with query IDs > - > > Key: IMPALA-6665 > URL: https://issues.apache.org/jira/browse/IMPALA-6665 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Affects Versions: Impala 2.12.0 >Reporter: bharath v >Assignee: Zoram Thanga >Priority: Major > Labels: supportability > > Similar to IMPALA-6664. The idea is to improve catalog server logging by > adding query-ID to each of the Catalog server log statements. This helps map > Catalog errors to specific queries, which is currently not possible. > Raising a separate jira for the Catalog server since fixing it could be a > little tricker than the other components since we don't have the query hash > readily available in the Catalog context. We need to augment the Catalog RPCs > with this data. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6664) Tag log statements with query-ids
[ https://issues.apache.org/jira/browse/IMPALA-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga reassigned IMPALA-6664: Assignee: Zoram Thanga > Tag log statements with query-ids > - > > Key: IMPALA-6664 > URL: https://issues.apache.org/jira/browse/IMPALA-6664 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Affects Versions: Impala 2.12.0 >Reporter: bharath v >Assignee: Zoram Thanga >Priority: Major > Labels: supportability > > The idea is to include query-id in each logged statement in > coordinator/finsts/planner. This helps to collect query specific logs by just > grep'ing through the logs with the query-id. > This is hopefully easier to implement, now that we have the query debug info > available in the thread context [1]. > [1] https://issues.apache.org/jira/browse/IMPALA-3703 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-4065) Inline comparator calls into TopN::InsertBatch()
[ https://issues.apache.org/jira/browse/IMPALA-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505508#comment-16505508 ] Zoram Thanga commented on IMPALA-4065: -- Assigned you the jira, [~tianyiwang]. > Inline comparator calls into TopN::InsertBatch() > > > Key: IMPALA-4065 > URL: https://issues.apache.org/jira/browse/IMPALA-4065 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.7.0 >Reporter: Tim Armstrong >Assignee: Tianyi Wang >Priority: Minor > Labels: codegen, ramp-up > > This is the more interesting follow-on from IMPALA-3815. We should inline the > Compare() calls in the codegen'd TopN code to avoid the indirect function > pointer call. > The tricky aspect is that the Compare() calls are called from > std::priority_queue, and we don't have a way to force-inline those functions > at the moment. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-4065) Inline comparator calls into TopN::InsertBatch()
[ https://issues.apache.org/jira/browse/IMPALA-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga reassigned IMPALA-4065: Assignee: Tianyi Wang (was: Zoram Thanga) > Inline comparator calls into TopN::InsertBatch() > > > Key: IMPALA-4065 > URL: https://issues.apache.org/jira/browse/IMPALA-4065 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.7.0 >Reporter: Tim Armstrong >Assignee: Tianyi Wang >Priority: Minor > Labels: codegen, ramp-up > > This is the more interesting follow-on from IMPALA-3815. We should inline the > Compare() calls in the codegen'd TopN code to avoid the indirect function > pointer call. > The tricky aspect is that the Compare() calls are called from > std::priority_queue, and we don't have a way to force-inline those functions > at the moment. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6086) Use of permanent function should require SELECT privilege on DB
[
https://issues.apache.org/jira/browse/IMPALA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495564#comment-16495564
]
Zoram Thanga commented on IMPALA-6086:
--
I just checked on a 2.7 cluster, and it seems like the issue of a
non-privileged user being able to execute UDF existed there as well:
{quote}
[nightly59-4.gce.cloudera.com:21000] > show tables;
Query: show tables
ERROR: AuthorizationException: User 'y...@gce.cloudera.com' does not have
privileges to access: default.*
[nightly59-4.gce.cloudera.com:21000] > select trim(' abcd ');
Query: select trim(' abcd ')
Query submitted at: 2018-05-30 11:58:24 (Coordinator:
https://nightly59-4.gce.cloudera.com:25000)
Query progress can be monitored at:
https://nightly59-4.gce.cloudera.com:25000/query_plan?query_id=58492b5157c2c36a:eeb57b5b
++
| trim(' abcd ') |
++
| abcd |
++
Fetched 1 row(s) in 0.01s
{quote}
> Use of permanent function should require SELECT privilege on DB
> ---
>
> Key: IMPALA-6086
> URL: https://issues.apache.org/jira/browse/IMPALA-6086
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Security
>Affects Versions: Impala 2.9.0, Impala 3.1.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Minor
>
> A user that has no privilege on a database should not be able to execute any
> permanent functions in that database. This is currently possible, and should
> be fixed, so that the user must have SELECT privilege to execute permanent
> functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-6086) Use of permanent function should require SELECT privilege on DB
[ https://issues.apache.org/jira/browse/IMPALA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga updated IMPALA-6086: - Affects Version/s: Impala 3.1.0 > Use of permanent function should require SELECT privilege on DB > --- > > Key: IMPALA-6086 > URL: https://issues.apache.org/jira/browse/IMPALA-6086 > Project: IMPALA > Issue Type: Bug > Components: Catalog, Security >Affects Versions: Impala 2.9.0, Impala 3.1.0 >Reporter: Zoram Thanga >Assignee: Zoram Thanga >Priority: Minor > > A user that has no privilege on a database should not be able to execute any > permanent functions in that database. This is currently possible, and should > be fixed, so that the user must have SELECT privilege to execute permanent > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6086) Use of permanent function should require SELECT privilege on DB
[
https://issues.apache.org/jira/browse/IMPALA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494461#comment-16494461
]
Zoram Thanga commented on IMPALA-6086:
--
The issue seems to be that the function call "trim('abcd')" gets
re-written/constant-folded into a string literal. We don't require any
privileges for literals, hence the regression(?). I haven't actually tested
this on an older release, but IMPALA-4574 and IMPALA-4586 touch on related
issues.
[~tarmstr...@cloudera.com], does this issue look familiar?
> Use of permanent function should require SELECT privilege on DB
> ---
>
> Key: IMPALA-6086
> URL: https://issues.apache.org/jira/browse/IMPALA-6086
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Security
>Affects Versions: Impala 2.9.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Minor
>
> A user that has no privilege on a database should not be able to execute any
> permanent functions in that database. This is currently possible, and should
> be fixed, so that the user must have SELECT privilege to execute permanent
> functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6086) Use of permanent function should require SELECT privilege on DB
[
https://issues.apache.org/jira/browse/IMPALA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494379#comment-16494379
]
Zoram Thanga commented on IMPALA-6086:
--
Finally had some cycles to dig into this. Looks like expression rewrite may be
the culprit, as we seem to be losing the privilegeReqs on re-analysis after the
rewrite. Here's a sample interaction on a Sentry-enabled Impala:
{quote}
[localhost:21000] default> show tables;
Query: show tables
ERROR: AuthorizationException: User 'zoram' does not have privileges to access:
default.*.*
[localhost:21000] default> select trim('abcd');
Query: select trim('abcd')
Query submitted at: 2018-05-29 15:22:47 (Coordinator:
http://zoram-desktop:25000)
Query progress can be monitored at:
http://zoram-desktop:25000/query_plan?query_id=3f48cb729a94afd4:6692d423
+--+
| trim('abcd') |
+--+
| abcd |
+--+
Fetched 1 row(s) in 4.91s
[localhost:21000] default> set ENABLE_EXPR_REWRITES = FALSE;
ENABLE_EXPR_REWRITES set to FALSE
[localhost:21000] default> select trim('abcd');
Query: select trim('abcd')
Query submitted at: 2018-05-29 15:23:07 (Coordinator:
http://zoram-desktop:25000)
ERROR: AuthorizationException: Cannot modify system database.
[localhost:21000] default>
{quote}
> Use of permanent function should require SELECT privilege on DB
> ---
>
> Key: IMPALA-6086
> URL: https://issues.apache.org/jira/browse/IMPALA-6086
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Security
>Affects Versions: Impala 2.9.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Minor
>
> A user that has no privilege on a database should not be able to execute any
> permanent functions in that database. This is currently possible, and should
> be fixed, so that the user must have SELECT privilege to execute permanent
> functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-6996) PartitionedAggregationNode::Close() should not dump stack trace
[
https://issues.apache.org/jira/browse/IMPALA-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga resolved IMPALA-6996.
--
Resolution: Duplicate
> PartitionedAggregationNode::Close() should not dump stack trace
> ---
>
> Key: IMPALA-6996
> URL: https://issues.apache.org/jira/browse/IMPALA-6996
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.12.0
>Reporter: Zoram Thanga
>Priority: Critical
>
> A CTAS query with MEM_LIMIT=6GB hit the limit while in an AggregationNode.
> Upon processing the failure, the fragment instance that hit the MEM_LIMIT
> tries to allocate more memory, and repeatedly dumps the following error
> message and stack trace to Impalad log file, once per row.
> {noformat}
> I0508 01:32:10.611563 40614 status.cc:55] Memory limit exceeded:
> FunctionContextImpl::AllocateLocal's allocations exceeded memory limits.
> Exprs could not allocate 3.00 B without exceeding limit.
> Error occurred on backend .com:22000 by fragment
> b429287bdabe184:9e7b5e96002c
> Memory left in process limit: 384.47 GB
> Memory left in query limit: -6947009.00 B
> Query(b429287bdabe184:9e7b5e96): memory limit exceeded. Limit=6.00 GB
> Total=6.01 GB Peak=6.01 GB
> Fragment b429287bdabe184:9e7b5e96002c: Total=2.76 GB Peak=2.76 GB
> AGGREGATION_NODE (id=3): Total=2.75 GB Peak=2.75 GB
> Exprs: Total=751.94 MB Peak=751.94 MB
> EXCHANGE_NODE (id=2): Total=0 Peak=0
> DataStreamRecvr: Total=7.42 MB Peak=7.42 MB
> CodeGen: Total=5.57 KB Peak=745.50 KB
> Block Manager: Limit=4.80 GB Total=4.39 GB Peak=4.40 GB
> Fragment b429287bdabe184:9e7b5e96000c: Total=3.25 GB Peak=3.25 GB
> AGGREGATION_NODE (id=1): Total=3.19 GB Peak=3.19 GB
> Exprs: Total=816.06 MB Peak=816.06 MB
> HDFS_SCAN_NODE (id=0): Total=59.96 MB Peak=200.09 MB
> CodeGen: Total=5.96 KB Peak=876.00 KB
> @ 0x83c78a impala::Status::Status()
> @ 0x83c98e impala::Status::MemLimitExceeded()
> @ 0xa24344 impala::MemTracker::MemLimitExceeded()
> @ 0xa35ccd impala::RuntimeState::SetMemLimitExceeded()
> @ 0xb6bd6d impala::FunctionContextImpl::CheckAllocResult()
> @ 0xb6ae78 impala::FunctionContextImpl::AllocateLocal()
> @ 0xb6b10f impala_udf::StringVal::StringVal()
> @ 0xb6b16a impala_udf::StringVal::CopyFrom()
> @ 0x8a2641 impala::AggregateFunctions::StringValGetValue()
> @ 0x8a2661
> impala::AggregateFunctions::StringValSerializeOrFinalize()
> @ 0xd411c5 impala::AggFnEvaluator::SerializeOrFinalize()
> @ 0xce6569 impala::PartitionedAggregationNode::CleanupHashTbl()
> @ 0xce693b
> impala::PartitionedAggregationNode::Partition::Close()
> @ 0xce8152
> impala::PartitionedAggregationNode::ClosePartitions()
> @ 0xcf163e impala::PartitionedAggregationNode::Close()
> @ 0xa7ae35 impala::FragmentInstanceState::Close()
> @ 0xa7ebfb impala::FragmentInstanceState::Exec()
> @ 0xa6aaf6 impala::QueryState::ExecFInstance()
> @ 0xbef9d9 impala::Thread::SuperviseThread()
> @ 0xbf0394 boost::detail::thread_data<>::run()
> @ 0xe588aa (unknown)
> @ 0x7f8a54c32e25 start_thread
> @ 0x7f8a5496034d __clone
> {noformat}
> Here't the profile summary:
> {noformat}
> Operator #Hosts Avg Time Max Time#Rows Est. #Rows Peak Mem
> Est. Peak Mem Detail
> ---
> 03:AGGREGATE 32 0.000ns 0.000ns0 1.06B2.75 GB
> 165.23 GB FINALIZE
> 02:EXCHANGE32 1s246ms 7s429ms 412.85M 1.06B 0
> 0 HASH()
> 01:AGGREGATE 32 22s539ms 27s631ms 422.16M 1.06B3.09 GB
> 165.23 GB STREAMING
> 00:SCAN HDFS 32 1s221ms 2s093ms 810.12M 1.06B 222.22 MB
> 616.00 MB
> Errors: Memory limit exceeded: Error occurred on backend
> .com:22000 by fragment b429287bdabe184:9e7b5e96002c
> Memory left in process limit: 386.76 GB
> Memory left in query limit: -19080.00 B
> {noformat}
> And the query string:
> {noformat}
> create TABLE TT>
> STORED AS PARQUET
> TBLPROPERTIES('parquet.compression'='SNAPPY')
> T>AS
> T>SELECT T>,
> ,
> ,
> MAX() AS ,
> MAX() AS ,
> MAX() AS ,
>
[jira] [Commented] (IMPALA-6996) PartitionedAggregationNode::Close() should not dump stack trace
[
https://issues.apache.org/jira/browse/IMPALA-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471506#comment-16471506
]
Zoram Thanga commented on IMPALA-6996:
--
Dup'ing this to IMPALA-6997, as the narrowed scope of this is essentially the
same.
> PartitionedAggregationNode::Close() should not dump stack trace
> ---
>
> Key: IMPALA-6996
> URL: https://issues.apache.org/jira/browse/IMPALA-6996
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.12.0
>Reporter: Zoram Thanga
>Priority: Critical
>
> A CTAS query with MEM_LIMIT=6GB hit the limit while in an AggregationNode.
> Upon processing the failure, the fragment instance that hit the MEM_LIMIT
> tries to allocate more memory, and repeatedly dumps the following error
> message and stack trace to Impalad log file, once per row.
> {noformat}
> I0508 01:32:10.611563 40614 status.cc:55] Memory limit exceeded:
> FunctionContextImpl::AllocateLocal's allocations exceeded memory limits.
> Exprs could not allocate 3.00 B without exceeding limit.
> Error occurred on backend .com:22000 by fragment
> b429287bdabe184:9e7b5e96002c
> Memory left in process limit: 384.47 GB
> Memory left in query limit: -6947009.00 B
> Query(b429287bdabe184:9e7b5e96): memory limit exceeded. Limit=6.00 GB
> Total=6.01 GB Peak=6.01 GB
> Fragment b429287bdabe184:9e7b5e96002c: Total=2.76 GB Peak=2.76 GB
> AGGREGATION_NODE (id=3): Total=2.75 GB Peak=2.75 GB
> Exprs: Total=751.94 MB Peak=751.94 MB
> EXCHANGE_NODE (id=2): Total=0 Peak=0
> DataStreamRecvr: Total=7.42 MB Peak=7.42 MB
> CodeGen: Total=5.57 KB Peak=745.50 KB
> Block Manager: Limit=4.80 GB Total=4.39 GB Peak=4.40 GB
> Fragment b429287bdabe184:9e7b5e96000c: Total=3.25 GB Peak=3.25 GB
> AGGREGATION_NODE (id=1): Total=3.19 GB Peak=3.19 GB
> Exprs: Total=816.06 MB Peak=816.06 MB
> HDFS_SCAN_NODE (id=0): Total=59.96 MB Peak=200.09 MB
> CodeGen: Total=5.96 KB Peak=876.00 KB
> @ 0x83c78a impala::Status::Status()
> @ 0x83c98e impala::Status::MemLimitExceeded()
> @ 0xa24344 impala::MemTracker::MemLimitExceeded()
> @ 0xa35ccd impala::RuntimeState::SetMemLimitExceeded()
> @ 0xb6bd6d impala::FunctionContextImpl::CheckAllocResult()
> @ 0xb6ae78 impala::FunctionContextImpl::AllocateLocal()
> @ 0xb6b10f impala_udf::StringVal::StringVal()
> @ 0xb6b16a impala_udf::StringVal::CopyFrom()
> @ 0x8a2641 impala::AggregateFunctions::StringValGetValue()
> @ 0x8a2661
> impala::AggregateFunctions::StringValSerializeOrFinalize()
> @ 0xd411c5 impala::AggFnEvaluator::SerializeOrFinalize()
> @ 0xce6569 impala::PartitionedAggregationNode::CleanupHashTbl()
> @ 0xce693b
> impala::PartitionedAggregationNode::Partition::Close()
> @ 0xce8152
> impala::PartitionedAggregationNode::ClosePartitions()
> @ 0xcf163e impala::PartitionedAggregationNode::Close()
> @ 0xa7ae35 impala::FragmentInstanceState::Close()
> @ 0xa7ebfb impala::FragmentInstanceState::Exec()
> @ 0xa6aaf6 impala::QueryState::ExecFInstance()
> @ 0xbef9d9 impala::Thread::SuperviseThread()
> @ 0xbf0394 boost::detail::thread_data<>::run()
> @ 0xe588aa (unknown)
> @ 0x7f8a54c32e25 start_thread
> @ 0x7f8a5496034d __clone
> {noformat}
> Here't the profile summary:
> {noformat}
> Operator #Hosts Avg Time Max Time#Rows Est. #Rows Peak Mem
> Est. Peak Mem Detail
> ---
> 03:AGGREGATE 32 0.000ns 0.000ns0 1.06B2.75 GB
> 165.23 GB FINALIZE
> 02:EXCHANGE32 1s246ms 7s429ms 412.85M 1.06B 0
> 0 HASH()
> 01:AGGREGATE 32 22s539ms 27s631ms 422.16M 1.06B3.09 GB
> 165.23 GB STREAMING
> 00:SCAN HDFS 32 1s221ms 2s093ms 810.12M 1.06B 222.22 MB
> 616.00 MB
> Errors: Memory limit exceeded: Error occurred on backend
> .com:22000 by fragment b429287bdabe184:9e7b5e96002c
> Memory left in process limit: 386.76 GB
> Memory left in query limit: -19080.00 B
> {noformat}
> And the query string:
> {noformat}
> create TABLE TT>
> STORED AS PARQUET
> TBLPROPERTIES('parquet.compression'='SNAPPY')
> T>AS
> T>SELECT T>,
> ,
> ,
> MAX()
[jira] [Commented] (IMPALA-6764) Codegend UnionNode::MaterializeBatch() causes memory corruption crash of Impalad
[
https://issues.apache.org/jira/browse/IMPALA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470966#comment-16470966
]
Zoram Thanga commented on IMPALA-6764:
--
[~dhecht]:
>> Do we have an example of the bad IR to x86 generation?
Attaching the assembly code of UnionNode::MaterializeBatch() for both cases.
The only change from 'bad' to 'good' is this change to StringFunctions::Trim():
{code:java}
$ git diff
diff --git a/be/src/exprs/string-functions-ir.cc
b/be/src/exprs/string-functions-ir.cc
index 49bc8c1..88af5d8 100644
--- a/be/src/exprs/string-functions-ir.cc
+++ b/be/src/exprs/string-functions-ir.cc
@@ -409,7 +409,7 @@ StringVal StringFunctions::Trim(FunctionContext* context,
const StringVal& str)
}
// Find new ending position.
int32_t end = str.len - 1;
- while (end > begin && str.ptr[end] == ' ') {
+ while (end >= begin && str.ptr[end] == ' ') {
--end;
}
return StringVal(str.ptr + begin, end - begin + 1);
{code}
>> Is there a later LLVM 3.9.x release that has the fix?
I am not sure when exactly the fix came in. FYI, there's no 3.9.x where x > 1.
LLVM changed their release taxonomy after 3.9.1. See
[http://releases.llvm.org/|http://releases.llvm.org/]. My 'wild guess' is that
the issue may have been fixed in 5.0.0.
> Codegend UnionNode::MaterializeBatch() causes memory corruption crash of
> Impalad
>
>
> Key: IMPALA-6764
> URL: https://issues.apache.org/jira/browse/IMPALA-6764
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Critical
>
> A CTAS statement involving UNION ALL with LEFT JOIN children is reliably
> crashing with a stack trace similar to the following:
> {noformat}
> (gdb) bt
> #0 0x7fb85fdf11f7 in raise () from ./debug-stuff/lib64/libc.so.6
> #1 0x7fb85fdf28e8 in abort () from ./debug-stuff/lib64/libc.so.6
> #2 0x7fb862106f35 in os::abort(bool) () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #3 0x7fb8622aaf33 in VMError::report_and_die() () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #4 0x7fb86210d22f in JVM_handle_linux_signal () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #5 0x7fb862103253 in signalHandler(int, siginfo*, void*) () from
> ./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x7fb85ff08706 in __memcpy_ssse3_back () from
> ./debug-stuff/lib64/libc.so.6
> #8 0x7fb840700d73 in
> impala::UnionNode::MaterializeBatch(impala::RowBatch*, unsigned char**)
> [clone .588] ()
> #9 0x01001806 in impala::UnionNode::GetNextMaterialized
> (this=this@entry=0x828, state=state@entry=0x848ed00,
> row_batch=row_batch@entry=0xcef9950)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:228
> #10 0x01001b5c in impala::UnionNode::GetNext (this=0x828,
> state=0x848ed00, row_batch=0xcef9950, eos=0x7fb7fe9a987e)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:294
> #11 0x00b724d2 in impala::FragmentInstanceState::ExecInternal
> (this=this@entry=0x4c030c0)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:270
> #12 0x00b74e42 in impala::FragmentInstanceState::Exec
> (this=this@entry=0x4c030c0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:89
> #13 0x00b64488 in impala::QueryState::ExecFInstance (this=0x8559200,
> fis=0x4c030c0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/query-state.cc:382
> #14 0x00d13613 in boost::function0::operator()
> (this=0x7fb7fe9a9c60)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #15 impala::Thread::SuperviseThread(std::string const&, std::string const&,
> boost::function, impala::Promise*) (name=..., category=...,
> functor=...,
> thread_started=0x7fb7f999f0f0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/util/thread.cc:352
> #16 0x00d13d54 in
> boost::_bi::list4 std::char_traits, std::allocator > >,
> boost::_bi::value std::allocator > >, boost::_bi::value >,
> boost::_bi::value >::operator() std::basic_string&, const std::basic_string&,
> boost::function, impala::Promise*), boost::_bi::list0> (
> f=@0x808bfb8: 0xd13460 const&, std::string const&, boost::function,
> impala::Promise*)>, a=,
> this=0x808bfc0)
[jira] [Updated] (IMPALA-6996) PartitionedAggregationNode::Close() should not dump stack trace
[
https://issues.apache.org/jira/browse/IMPALA-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga updated IMPALA-6996:
-
Summary: PartitionedAggregationNode::Close() should not dump stack trace
(was: PartitionedAggregationNode::Close() should not allocate memory)
> PartitionedAggregationNode::Close() should not dump stack trace
> ---
>
> Key: IMPALA-6996
> URL: https://issues.apache.org/jira/browse/IMPALA-6996
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.12.0
>Reporter: Zoram Thanga
>Priority: Critical
>
> A CTAS query with MEM_LIMIT=6GB hit the limit while in an AggregationNode.
> Upon processing the failure, the fragment instance that hit the MEM_LIMIT
> tries to allocate more memory, and repeatedly dumps the following error
> message and stack trace to Impalad log file, once per row.
> {noformat}
> I0508 01:32:10.611563 40614 status.cc:55] Memory limit exceeded:
> FunctionContextImpl::AllocateLocal's allocations exceeded memory limits.
> Exprs could not allocate 3.00 B without exceeding limit.
> Error occurred on backend .com:22000 by fragment
> b429287bdabe184:9e7b5e96002c
> Memory left in process limit: 384.47 GB
> Memory left in query limit: -6947009.00 B
> Query(b429287bdabe184:9e7b5e96): memory limit exceeded. Limit=6.00 GB
> Total=6.01 GB Peak=6.01 GB
> Fragment b429287bdabe184:9e7b5e96002c: Total=2.76 GB Peak=2.76 GB
> AGGREGATION_NODE (id=3): Total=2.75 GB Peak=2.75 GB
> Exprs: Total=751.94 MB Peak=751.94 MB
> EXCHANGE_NODE (id=2): Total=0 Peak=0
> DataStreamRecvr: Total=7.42 MB Peak=7.42 MB
> CodeGen: Total=5.57 KB Peak=745.50 KB
> Block Manager: Limit=4.80 GB Total=4.39 GB Peak=4.40 GB
> Fragment b429287bdabe184:9e7b5e96000c: Total=3.25 GB Peak=3.25 GB
> AGGREGATION_NODE (id=1): Total=3.19 GB Peak=3.19 GB
> Exprs: Total=816.06 MB Peak=816.06 MB
> HDFS_SCAN_NODE (id=0): Total=59.96 MB Peak=200.09 MB
> CodeGen: Total=5.96 KB Peak=876.00 KB
> @ 0x83c78a impala::Status::Status()
> @ 0x83c98e impala::Status::MemLimitExceeded()
> @ 0xa24344 impala::MemTracker::MemLimitExceeded()
> @ 0xa35ccd impala::RuntimeState::SetMemLimitExceeded()
> @ 0xb6bd6d impala::FunctionContextImpl::CheckAllocResult()
> @ 0xb6ae78 impala::FunctionContextImpl::AllocateLocal()
> @ 0xb6b10f impala_udf::StringVal::StringVal()
> @ 0xb6b16a impala_udf::StringVal::CopyFrom()
> @ 0x8a2641 impala::AggregateFunctions::StringValGetValue()
> @ 0x8a2661
> impala::AggregateFunctions::StringValSerializeOrFinalize()
> @ 0xd411c5 impala::AggFnEvaluator::SerializeOrFinalize()
> @ 0xce6569 impala::PartitionedAggregationNode::CleanupHashTbl()
> @ 0xce693b
> impala::PartitionedAggregationNode::Partition::Close()
> @ 0xce8152
> impala::PartitionedAggregationNode::ClosePartitions()
> @ 0xcf163e impala::PartitionedAggregationNode::Close()
> @ 0xa7ae35 impala::FragmentInstanceState::Close()
> @ 0xa7ebfb impala::FragmentInstanceState::Exec()
> @ 0xa6aaf6 impala::QueryState::ExecFInstance()
> @ 0xbef9d9 impala::Thread::SuperviseThread()
> @ 0xbf0394 boost::detail::thread_data<>::run()
> @ 0xe588aa (unknown)
> @ 0x7f8a54c32e25 start_thread
> @ 0x7f8a5496034d __clone
> {noformat}
> Here't the profile summary:
> {noformat}
> Operator #Hosts Avg Time Max Time#Rows Est. #Rows Peak Mem
> Est. Peak Mem Detail
> ---
> 03:AGGREGATE 32 0.000ns 0.000ns0 1.06B2.75 GB
> 165.23 GB FINALIZE
> 02:EXCHANGE32 1s246ms 7s429ms 412.85M 1.06B 0
> 0 HASH()
> 01:AGGREGATE 32 22s539ms 27s631ms 422.16M 1.06B3.09 GB
> 165.23 GB STREAMING
> 00:SCAN HDFS 32 1s221ms 2s093ms 810.12M 1.06B 222.22 MB
> 616.00 MB
> Errors: Memory limit exceeded: Error occurred on backend
> .com:22000 by fragment b429287bdabe184:9e7b5e96002c
> Memory left in process limit: 386.76 GB
> Memory left in query limit: -19080.00 B
> {noformat}
> And the query string:
> {noformat}
> create TABLE TT>
> STORED AS PARQUET
> TBLPROPERTIES('parquet.compression'='SNAPPY')
> T>AS
> T>SELECT T>,
> ,
> ,
>
[jira] [Created] (IMPALA-6996) PartitionedAggregationNode::Close() should not allocate memory
Zoram Thanga created IMPALA-6996:
Summary: PartitionedAggregationNode::Close() should not allocate
memory
Key: IMPALA-6996
URL: https://issues.apache.org/jira/browse/IMPALA-6996
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.12.0
Reporter: Zoram Thanga
A CTAS query with MEM_LIMIT=6GB hit the limit while in an AggregationNode. Upon
processing the failure, the fragment instance that hit the MEM_LIMIT tries to
allocate more memory, and repeatedly dumps the following error message and
stack trace to Impalad log file, once per row.
{noformat}
I0508 01:32:10.611563 40614 status.cc:55] Memory limit exceeded:
FunctionContextImpl::AllocateLocal's allocations exceeded memory limits.
Exprs could not allocate 3.00 B without exceeding limit.
Error occurred on backend .com:22000 by fragment
b429287bdabe184:9e7b5e96002c
Memory left in process limit: 384.47 GB
Memory left in query limit: -6947009.00 B
Query(b429287bdabe184:9e7b5e96): memory limit exceeded. Limit=6.00 GB
Total=6.01 GB Peak=6.01 GB
Fragment b429287bdabe184:9e7b5e96002c: Total=2.76 GB Peak=2.76 GB
AGGREGATION_NODE (id=3): Total=2.75 GB Peak=2.75 GB
Exprs: Total=751.94 MB Peak=751.94 MB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=7.42 MB Peak=7.42 MB
CodeGen: Total=5.57 KB Peak=745.50 KB
Block Manager: Limit=4.80 GB Total=4.39 GB Peak=4.40 GB
Fragment b429287bdabe184:9e7b5e96000c: Total=3.25 GB Peak=3.25 GB
AGGREGATION_NODE (id=1): Total=3.19 GB Peak=3.19 GB
Exprs: Total=816.06 MB Peak=816.06 MB
HDFS_SCAN_NODE (id=0): Total=59.96 MB Peak=200.09 MB
CodeGen: Total=5.96 KB Peak=876.00 KB
@ 0x83c78a impala::Status::Status()
@ 0x83c98e impala::Status::MemLimitExceeded()
@ 0xa24344 impala::MemTracker::MemLimitExceeded()
@ 0xa35ccd impala::RuntimeState::SetMemLimitExceeded()
@ 0xb6bd6d impala::FunctionContextImpl::CheckAllocResult()
@ 0xb6ae78 impala::FunctionContextImpl::AllocateLocal()
@ 0xb6b10f impala_udf::StringVal::StringVal()
@ 0xb6b16a impala_udf::StringVal::CopyFrom()
@ 0x8a2641 impala::AggregateFunctions::StringValGetValue()
@ 0x8a2661
impala::AggregateFunctions::StringValSerializeOrFinalize()
@ 0xd411c5 impala::AggFnEvaluator::SerializeOrFinalize()
@ 0xce6569 impala::PartitionedAggregationNode::CleanupHashTbl()
@ 0xce693b impala::PartitionedAggregationNode::Partition::Close()
@ 0xce8152 impala::PartitionedAggregationNode::ClosePartitions()
@ 0xcf163e impala::PartitionedAggregationNode::Close()
@ 0xa7ae35 impala::FragmentInstanceState::Close()
@ 0xa7ebfb impala::FragmentInstanceState::Exec()
@ 0xa6aaf6 impala::QueryState::ExecFInstance()
@ 0xbef9d9 impala::Thread::SuperviseThread()
@ 0xbf0394 boost::detail::thread_data<>::run()
@ 0xe588aa (unknown)
@ 0x7f8a54c32e25 start_thread
@ 0x7f8a5496034d __clone
{noformat}
Here't the profile summary:
{noformat}
Operator #Hosts Avg Time Max Time#Rows Est. #Rows Peak Mem Est.
Peak Mem Detail
---
03:AGGREGATE 32 0.000ns 0.000ns0 1.06B2.75 GB
165.23 GB FINALIZE
02:EXCHANGE32 1s246ms 7s429ms 412.85M 1.06B 0
0 HASH()
01:AGGREGATE 32 22s539ms 27s631ms 422.16M 1.06B3.09 GB
165.23 GB STREAMING
00:SCAN HDFS 32 1s221ms 2s093ms 810.12M 1.06B 222.22 MB
616.00 MB
Errors: Memory limit exceeded: Error occurred on backend
.com:22000 by fragment b429287bdabe184:9e7b5e96002c
Memory left in process limit: 386.76 GB
Memory left in query limit: -19080.00 B
{noformat}
And the query string:
{noformat}
create TABLE TT>
STORED AS PARQUET
TBLPROPERTIES('parquet.compression'='SNAPPY')
T>AS
T>SELECT T>,
,
,
MAX() AS ,
MAX() AS ,
MAX() AS ,
MAX() AS
T>FROM TT>
GROUP BY,
,
Coordinator: .com:22000
Query Options (non default):
MEM_LIMIT=6442450944,REQUEST_POOL=,SYNC_DDL=1,MT_DOP=0
DDL Type: CREATE_TABLE_AS_SELECT
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-6996) PartitionedAggregationNode::Close() should not allocate memory
Zoram Thanga created IMPALA-6996:
Summary: PartitionedAggregationNode::Close() should not allocate
memory
Key: IMPALA-6996
URL: https://issues.apache.org/jira/browse/IMPALA-6996
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.12.0
Reporter: Zoram Thanga
A CTAS query with MEM_LIMIT=6GB hit the limit while in an AggregationNode. Upon
processing the failure, the fragment instance that hit the MEM_LIMIT tries to
allocate more memory, and repeatedly dumps the following error message and
stack trace to Impalad log file, once per row.
{noformat}
I0508 01:32:10.611563 40614 status.cc:55] Memory limit exceeded:
FunctionContextImpl::AllocateLocal's allocations exceeded memory limits.
Exprs could not allocate 3.00 B without exceeding limit.
Error occurred on backend .com:22000 by fragment
b429287bdabe184:9e7b5e96002c
Memory left in process limit: 384.47 GB
Memory left in query limit: -6947009.00 B
Query(b429287bdabe184:9e7b5e96): memory limit exceeded. Limit=6.00 GB
Total=6.01 GB Peak=6.01 GB
Fragment b429287bdabe184:9e7b5e96002c: Total=2.76 GB Peak=2.76 GB
AGGREGATION_NODE (id=3): Total=2.75 GB Peak=2.75 GB
Exprs: Total=751.94 MB Peak=751.94 MB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=7.42 MB Peak=7.42 MB
CodeGen: Total=5.57 KB Peak=745.50 KB
Block Manager: Limit=4.80 GB Total=4.39 GB Peak=4.40 GB
Fragment b429287bdabe184:9e7b5e96000c: Total=3.25 GB Peak=3.25 GB
AGGREGATION_NODE (id=1): Total=3.19 GB Peak=3.19 GB
Exprs: Total=816.06 MB Peak=816.06 MB
HDFS_SCAN_NODE (id=0): Total=59.96 MB Peak=200.09 MB
CodeGen: Total=5.96 KB Peak=876.00 KB
@ 0x83c78a impala::Status::Status()
@ 0x83c98e impala::Status::MemLimitExceeded()
@ 0xa24344 impala::MemTracker::MemLimitExceeded()
@ 0xa35ccd impala::RuntimeState::SetMemLimitExceeded()
@ 0xb6bd6d impala::FunctionContextImpl::CheckAllocResult()
@ 0xb6ae78 impala::FunctionContextImpl::AllocateLocal()
@ 0xb6b10f impala_udf::StringVal::StringVal()
@ 0xb6b16a impala_udf::StringVal::CopyFrom()
@ 0x8a2641 impala::AggregateFunctions::StringValGetValue()
@ 0x8a2661
impala::AggregateFunctions::StringValSerializeOrFinalize()
@ 0xd411c5 impala::AggFnEvaluator::SerializeOrFinalize()
@ 0xce6569 impala::PartitionedAggregationNode::CleanupHashTbl()
@ 0xce693b impala::PartitionedAggregationNode::Partition::Close()
@ 0xce8152 impala::PartitionedAggregationNode::ClosePartitions()
@ 0xcf163e impala::PartitionedAggregationNode::Close()
@ 0xa7ae35 impala::FragmentInstanceState::Close()
@ 0xa7ebfb impala::FragmentInstanceState::Exec()
@ 0xa6aaf6 impala::QueryState::ExecFInstance()
@ 0xbef9d9 impala::Thread::SuperviseThread()
@ 0xbf0394 boost::detail::thread_data<>::run()
@ 0xe588aa (unknown)
@ 0x7f8a54c32e25 start_thread
@ 0x7f8a5496034d __clone
{noformat}
Here't the profile summary:
{noformat}
Operator #Hosts Avg Time Max Time#Rows Est. #Rows Peak Mem Est.
Peak Mem Detail
---
03:AGGREGATE 32 0.000ns 0.000ns0 1.06B2.75 GB
165.23 GB FINALIZE
02:EXCHANGE32 1s246ms 7s429ms 412.85M 1.06B 0
0 HASH()
01:AGGREGATE 32 22s539ms 27s631ms 422.16M 1.06B3.09 GB
165.23 GB STREAMING
00:SCAN HDFS 32 1s221ms 2s093ms 810.12M 1.06B 222.22 MB
616.00 MB
Errors: Memory limit exceeded: Error occurred on backend
.com:22000 by fragment b429287bdabe184:9e7b5e96002c
Memory left in process limit: 386.76 GB
Memory left in query limit: -19080.00 B
{noformat}
And the query string:
{noformat}
create TABLE TT>
STORED AS PARQUET
TBLPROPERTIES('parquet.compression'='SNAPPY')
T>AS
T>SELECT T>,
,
,
MAX() AS ,
MAX() AS ,
MAX() AS ,
MAX() AS
T>FROM TT>
GROUP BY,
,
Coordinator: .com:22000
Query Options (non default):
MEM_LIMIT=6442450944,REQUEST_POOL=,SYNC_DDL=1,MT_DOP=0
DDL Type: CREATE_TABLE_AS_SELECT
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-6971) Log flooding with unhelpful parsing error message
Zoram Thanga created IMPALA-6971:
Summary: Log flooding with unhelpful parsing error message
Key: IMPALA-6971
URL: https://issues.apache.org/jira/browse/IMPALA-6971
Project: IMPALA
Issue Type: Improvement
Affects Versions: Impala 2.12.0, Impala 3.0
Reporter: Zoram Thanga
Once in a while I've seen Impala flood the log file with messages similar to
the following:
{noformat}
I0318 08:20:28.566603 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.566643 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.566676 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.566709 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.566742 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.566787 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.566820 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.566856 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.566890 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.566923 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.566959 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.566998 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.567039 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.567070 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.567101 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.567132 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.567173 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.567209 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.567245 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.567276 984564 runtime-state.cc:168] Error from query
cf41e4ee4f7e1273:bcc41038: Error parsing row: file:
hdfs://nameservice/user/hive/warehouse/mydb/mytable/part-m-0, before
offset: 2122317824
I0318 08:20:28.567312 984564
[jira] [Created] (IMPALA-6936) LLVM in aggressive optimization mode generates toxic x86 assembly for StringFunctions::Trim
Zoram Thanga created IMPALA-6936: Summary: LLVM in aggressive optimization mode generates toxic x86 assembly for StringFunctions::Trim Key: IMPALA-6936 URL: https://issues.apache.org/jira/browse/IMPALA-6936 Project: IMPALA Issue Type: Bug Components: Backend Affects Versions: Impala 2.11.0 Reporter: Zoram Thanga Assignee: Zoram Thanga Fix For: Impala 2.11.0 The aggressively optimized IR->x86 assembly translation for UnionNode::MaterializeBatch() generates code for StringFunctions::Trim() that causes a crash in memcpy() due to bad addresses being passed in. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-6930) Empty --llvm_cpu_attr_whitelist startup argument triggers crash in LLVM
Zoram Thanga created IMPALA-6930:
Summary: Empty --llvm_cpu_attr_whitelist startup argument triggers
crash in LLVM
Key: IMPALA-6930
URL: https://issues.apache.org/jira/browse/IMPALA-6930
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.11.0
Reporter: Zoram Thanga
While investigating IMPALA-6764, I tripped on the following crash after passing
--llvm_cpu_attr_whitelist='' startup option to impalad. We should probably
guard against this empty argument string. This happened with LLVM 3.9. I
haven't tried the experiment with the latest code, which has LLVM 5.0.1. It
looks like a DCHECK/CHECK kind of crash in LLVM:
{noformat}
(gdb) bt
#0 0x7f4821417428 in __GI_raise (sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:54
#1 0x7f482141902a in __GI_abort () at abort.c:89
#2 0x020ee124 in google::DumpStackTraceAndExit() ()
#3 0x020e4b9d in google::LogMessage::Fail() ()
#4 0x020e6442 in google::LogMessage::SendToLog() ()
#5 0x020e4577 in google::LogMessage::Flush() ()
#6 0x020e7b3e in google::LogMessageFatal::~LogMessageFatal() ()
#7 0x00d74f70 in impala::LlvmCodegenHandleError (user_data=, reason=..., gen_crash_diag=)
at /home/zoram/Impala/be/src/codegen/llvm-codegen.cc:126
#8 0x02023baa in llvm::report_fatal_error(llvm::Twine const&, bool) ()
#9 0x02023d4e in llvm::report_fatal_error(std::string const&, bool) ()
#10 0x01691c9a in
llvm::SelectionDAGISel::CannotYetSelect(llvm::SDNode*) ()
#11 0x016955bf in
llvm::SelectionDAGISel::SelectCodeCommon(llvm::SDNode*, unsigned char const*,
unsigned int) ()
#12 0x01494f8f in (anonymous
namespace)::X86DAGToDAGISel::Select(llvm::SDNode*) ()
#13 0x01690649 in llvm::SelectionDAGISel::DoInstructionSelection() ()
#14 0x01693301 in llvm::SelectionDAGISel::CodeGenAndEmitDAG() ()
#15 0x0169a674 in
llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) ()
#16 0x0169c4ff in
llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) ()
#17 0x01498424 in (anonymous
namespace)::X86DAGToDAGISel::runOnMachineFunction(llvm::MachineFunction&) ()
#18 0x018167d9 in
llvm::MachineFunctionPass::runOnFunction(llvm::Function&) ()
#19 0x01fac06a in llvm::FPPassManager::runOnFunction(llvm::Function&) ()
#20 0x01fac693 in llvm::legacy::PassManagerImpl::run(llvm::Module&) ()
#21 0x01c9d3a9 in llvm::MCJIT::emitObject(llvm::Module*) ()
#22 0x01c9d8b5 in llvm::MCJIT::generateCodeForModule(llvm::Module*) ()
#23 0x01c9a180 in llvm::MCJIT::finalizeObject() ()
#24 0x00d834c2 in impala::LlvmCodeGen::FinalizeModule (this=0xb73e580)
at /home/zoram/Impala/be/src/codegen/llvm-codegen.cc:1120
#25 0x00b8a7b8 in impala::FragmentInstanceState::Open
(this=this@entry=0x7ccf4a0) at
/home/zoram/Impala/be/src/runtime/fragment-instance-state.cc:254
#26 0x00b8bcdf in impala::FragmentInstanceState::Exec
(this=this@entry=0x7ccf4a0) at
/home/zoram/Impala/be/src/runtime/fragment-instance-state.cc:81
#27 0x00b7ad96 in impala::QueryState::ExecFInstance (this=0x67bcd00,
fis=0x7ccf4a0) at /home/zoram/Impala/be/src/runtime/query-state.cc:386
#28 0x00d3504f in boost::function0::operator()
(this=0x7f47bbae4d50)
at
/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
#29 impala::Thread::SuperviseThread(std::string const&, std::string const&,
boost::function, impala::Promise*) (name=..., category=...,
functor=...,
thread_started=) at
/home/zoram/Impala/be/src/util/thread.cc:354
#30 0x00d35843 in
boost::_bi::list4 >,
boost::_bi::value >, boost::_bi::value >,
boost::_bi::value >::operator()&, const std::basic_string&,
boost::function, impala::Promise*), boost::_bi::list0> (
f=@0x6b65db8: 0xd34dc0 ,
a=, this=0x6b65dc0) at
/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
#31 boost::_bi::bind_t,
boost::_bi::value > >::operator()() (this=0x6b65db8)
at
/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
#32 boost::detail::thread_data
[jira] [Created] (IMPALA-6895) Eliminate SimpleLogger flush threads by inlining flush
Zoram Thanga created IMPALA-6895: Summary: Eliminate SimpleLogger flush threads by inlining flush Key: IMPALA-6895 URL: https://issues.apache.org/jira/browse/IMPALA-6895 Project: IMPALA Issue Type: Improvement Components: Backend Affects Versions: Impala 3.0 Reporter: Zoram Thanga Currently, SimpleLogger provides a Flush() interface which is used by its client(s) to periodically (hard-coded to 5 seconds) flush the log file. We could eliminate these flush threads by keeping track of last flush time, and have the caller of SimpleLogger::AppendEntry() flush on demand (now - last_flush_time >= 5 seconds or whatever). This has the added benefit of reducing contention on the SimpleLogger::log_file_lock_ mutex to just between the threads adding entries to the log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-6764) Codegend UnionNode::MaterializeBatch() causes memory corruption crash of Impalad
Zoram Thanga created IMPALA-6764:
Summary: Codegend UnionNode::MaterializeBatch() causes memory
corruption crash of Impalad
Key: IMPALA-6764
URL: https://issues.apache.org/jira/browse/IMPALA-6764
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.11.0
Reporter: Zoram Thanga
Assignee: Zoram Thanga
A CTAS statement involving UNION ALL with LEFT JOIN children is reliably
crashing with a stack trace similar to the following:
{noformat}
(gdb) bt
#0 0x7fb85fdf11f7 in raise () from ./debug-stuff/lib64/libc.so.6
#1 0x7fb85fdf28e8 in abort () from ./debug-stuff/lib64/libc.so.6
#2 0x7fb862106f35 in os::abort(bool) () from
./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
#3 0x7fb8622aaf33 in VMError::report_and_die() () from
./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
#4 0x7fb86210d22f in JVM_handle_linux_signal () from
./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
#5 0x7fb862103253 in signalHandler(int, siginfo*, void*) () from
./debug-stuff/usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
#6
#7 0x7fb85ff08706 in __memcpy_ssse3_back () from
./debug-stuff/lib64/libc.so.6
#8 0x7fb840700d73 in
impala::UnionNode::MaterializeBatch(impala::RowBatch*, unsigned char**) [clone
.588] ()
#9 0x01001806 in impala::UnionNode::GetNextMaterialized
(this=this@entry=0x828, state=state@entry=0x848ed00,
row_batch=row_batch@entry=0xcef9950)
at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:228
#10 0x01001b5c in impala::UnionNode::GetNext (this=0x828,
state=0x848ed00, row_batch=0xcef9950, eos=0x7fb7fe9a987e)
at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/union-node.cc:294
#11 0x00b724d2 in impala::FragmentInstanceState::ExecInternal
(this=this@entry=0x4c030c0)
at
/usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:270
#12 0x00b74e42 in impala::FragmentInstanceState::Exec
(this=this@entry=0x4c030c0) at
/usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:89
#13 0x00b64488 in impala::QueryState::ExecFInstance (this=0x8559200,
fis=0x4c030c0) at
/usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/query-state.cc:382
#14 0x00d13613 in boost::function0::operator()
(this=0x7fb7fe9a9c60)
at
/usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
#15 impala::Thread::SuperviseThread(std::string const&, std::string const&,
boost::function, impala::Promise*) (name=..., category=...,
functor=...,
thread_started=0x7fb7f999f0f0) at
/usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/util/thread.cc:352
#16 0x00d13d54 in
boost::_bi::list4 >,
boost::_bi::value >, boost::_bi::value >,
boost::_bi::value >::operator()&, const std::basic_string&,
boost::function, impala::Promise*), boost::_bi::list0> (
f=@0x808bfb8: 0xd13460 ,
a=,
this=0x808bfc0) at
/usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
#17 boost::_bi::bind_t,
boost::_bi::value > >::operator()() (this=0x808bfb8)
at
/usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
#18 boost::detail::thread_data,
boost::_bi::value > > >::run() (this=0x808be00)
at
/usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
#19 0x0128e8ea in thread_proxy ()
#20 0x7fb860186e25 in start_thread () from
./debug-stuff/lib64/libpthread.so.0
#21 0x7fb85feb434d in clone () from ./debug-stuff/lib64/libc.so.6
{noformat}
The exact location or reason of the crash varies, i.e., sometimes we crash
while accessing the source address of memcpy, other times we crash on accessing
the destination address. In this particular instance, we see:
{noformat}
0x7fb85ff086e4 <+6676>: add%rdx,%rsi
0x7fb85ff086e7 <+6679>: add%rdx,%rdi
0x7fb85ff086ea <+6682>: lea0x375df(%rip),%r11
[jira] [Resolved] (IMPALA-6498) test_query_profile_thrift_timestamps causes following tests to fail
[ https://issues.apache.org/jira/browse/IMPALA-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga resolved IMPALA-6498. -- Resolution: Fixed Fix Version/s: Impala 2.12.0 Impala 3.0 > test_query_profile_thrift_timestamps causes following tests to fail > --- > > Key: IMPALA-6498 > URL: https://issues.apache.org/jira/browse/IMPALA-6498 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 2.12.0 >Reporter: Thomas Tauber-Marshall >Assignee: Zoram Thanga >Priority: Major > Fix For: Impala 3.0, Impala 2.12.0 > > > test_query_profile_thrift_timestamps calls "self.client.close()". This is not > really a valid thing for a test to do, as "self.client" is opened in > setup_class and expected to remain valid for all of the tests in a suite > until teardown_class is called, and so any tests following it in the class > that rely on "self.client" will not work. > This hasn't been a problem in builds yet as > test_query_profile_thrift_timestamps is marked "serial", there are no other > "serial" tests following it in the suite, and tests are normally run via > "tests/run-test.py", which runs the parallel and serial tests separately. > I only hit this because I was running tests via "bin/impala-py.test" which, > without any args, runs all tests serially, but we should still fix it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-6307) A CTAS query fails with error: AnalysisException: Duplicate column name:
[
https://issues.apache.org/jira/browse/IMPALA-6307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoram Thanga resolved IMPALA-6307.
--
Resolution: Fixed
Fix Version/s: Impala 2.12.0
> A CTAS query fails with error: AnalysisException: Duplicate column name:
>
> -
>
> Key: IMPALA-6307
> URL: https://issues.apache.org/jira/browse/IMPALA-6307
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.10.0, Impala 2.11.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Critical
> Labels: regression
> Fix For: Impala 2.12.0
>
>
> The following query triggers the exception:
> CREATE TABLE foo partitioned by (year) AS
> WITH TMP AS (
> SELECT a.timestamp_col, a.year FROM functional.alltypes a
> LEFT JOIN functional.alltypes b
> ON b.timestamp_col BETWEEN a.timestamp_col AND a.timestamp_col
> )
> SELECT a.timestamp_col, a.year FROM TMP a;
> The exception is thrown from TableDef::analyzeColumnDefs():
>
> {code:java}
> private void analyzeColumnDefs(Analyzer analyzer) throws AnalysisException {
> Set colNames = Sets.newHashSet();
> for (ColumnDef colDef: columnDefs_) {
> colDef.analyze(analyzer);
> if (!colNames.add(colDef.getColName().toLowerCase())) {
> throw new AnalysisException("Duplicate column name: " +
> colDef.getColName());
> }
> if (!isKuduTable() && colDef.hasKuduOptions()) {
> throw new AnalysisException(String.format("Unsupported column options
> for " +
> "file format '%s': '%s'", getFileFormat().name(),
> colDef.toString()));
> }
> }
> for (ColumnDef colDef: getPartitionColumnDefs()) {
> colDef.analyze(analyzer);
> if (!colDef.getType().supportsTablePartitioning()) {
> throw new AnalysisException(
> String.format("Type '%s' is not supported as partition-column
> type " +
> "in column: %s", colDef.getType().toSql(),
> colDef.getColName()));
> }
> if (!colNames.add(colDef.getColName().toLowerCase())) {
> throw new AnalysisException("Duplicate column name: " +
> colDef.getColName()); // THROWS HERE
> }
> }
> {code}
> The column duplication happens for "year" because it's in both columnDefs_
> and dataLayout_::partitionColDefs_.
> The issue does not reproduce is we replace BETWEEN in the JOIN clause with
> the equivalent "b.timestamp_col > a.timestamp_col AND b.timestamp_col <
> a.timestamp_col".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-6381) test_exchange_delays.py failed on isilon build due to sender timeout
Zoram Thanga created IMPALA-6381:
Summary: test_exchange_delays.py failed on isilon build due to
sender timeout
Key: IMPALA-6381
URL: https://issues.apache.org/jira/browse/IMPALA-6381
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 2.11.0
Reporter: Zoram Thanga
Assignee: Tim Armstrong
Please feel free to re-assign, etc.
Isilon builds have been failing because of this:
Error Message
AssertionError: Expected exception: Sender timed out waiting for receiver
fragment instance
Stacktrace
self =
vector =
@pytest.mark.execute_serially
@CustomClusterTestSuite.with_args(
"--stress_datastream_recvr_delay_ms={0}".format(DELAY_MS)
+ " --datastream_sender_timeout_ms=1")
def test_exchange_large_delay(self, vector):
"""Test delays in registering data stream receivers where all of the
batches sent
will time out before the receiver registers. Before IMPALA-2987, this
scenario
resulted in the query hanging.
"""
self.run_test_case('QueryTest/exchange-delays', vector)
# Test the special case when no batches are sent and the EOS message
times out.
> self.run_test_case('QueryTest/exchange-delays-zero-rows', vector)
custom_cluster/test_exchange_delays.py:58:
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (IMPALA-6225) IMPALA-5599 broke a client software due to precision change in date-time string
[ https://issues.apache.org/jira/browse/IMPALA-6225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga resolved IMPALA-6225. -- Resolution: Fixed Fix Version/s: Impala 2.11.0 > IMPALA-5599 broke a client software due to precision change in date-time > string > --- > > Key: IMPALA-6225 > URL: https://issues.apache.org/jira/browse/IMPALA-6225 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Zoram Thanga >Assignee: Zoram Thanga > Fix For: Impala 2.11.0 > > > IMPALA-5599 changed the precision of Start Time and End Time date-time > strings of query profiles to microseconds. This ended up breaking > compatibility with a management client software. > This JIRA is being raised to restore the precision to what it was > (nanoseconds) , to address the breakage. Longer term, the interface contract > between Impala and such client software will be made stricter and more > efficient, and avoid having clients parse what are meant to be human-readable > information. > The Impala testing also needs to be enhanced to ensure interface contracts > are not broken. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (IMPALA-6114) Incorrect type deduction causing analysis exception to be thrown
[ https://issues.apache.org/jira/browse/IMPALA-6114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoram Thanga resolved IMPALA-6114. -- Resolution: Fixed Fix Version/s: Impala 2.11.0 > Incorrect type deduction causing analysis exception to be thrown > > > Key: IMPALA-6114 > URL: https://issues.apache.org/jira/browse/IMPALA-6114 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 2.10.0, Impala 2.11.0 > Environment: Ubuntu 16.04, OS independent >Reporter: Zoram Thanga >Assignee: Zoram Thanga > Fix For: Impala 2.11.0 > > > The SELECT statement > SELECT DISTINCT CAST(0 AS DECIMAL(14)) AS foo, 0 AS bar FROM t; > fails with the following exception: > ERROR: AnalysisException: Possible loss of precision for target table > 'default.t'. > Expression '0' (type: DECIMAL(14,0)) would need to be cast to INT for column > 'bar' > The query succeeds if we remove the DISTINCT constraint. > The exception stack trace follows: > I1026 15:54:53.231101 17398 jni-util.cc:211] > org.apache.impala.common.AnalysisException: Possible loss of precision for > target table 'default.t'. > Expression '0' (type: DECIMAL(14,0)) would need to be cast to INT for column > 'bar' > at > org.apache.impala.analysis.StatementBase.checkTypeCompatibility(StatementBase.java:190) > at > org.apache.impala.analysis.InsertStmt.prepareExpressions(InsertStmt.java:674) > at org.apache.impala.analysis.InsertStmt.analyze(InsertStmt.java:383) > at > org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:418) > at > org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:369) > at org.apache.impala.service.Frontend.analyzeStmt(Frontend.java:900) > at > org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1049) > at > org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:156) > I1026 15:54:53.269991 17398 status.cc:122] AnalysisException: Possible loss > of precision for target table 'default.t'. > Expression '0' (type: DECIMAL(14,0)) would need to be cast to INT for column > 'bar' > @ 0x1598f53 impala::Status::Status() > @ 0x1993a5e impala::JniUtil::GetJniExceptionMsg() > @ 0x1843e50 impala::JniUtil::CallJniMethod<>() > @ 0x1840325 impala::Frontend::GetExecRequest() > @ 0x1860fca impala::ImpalaServer::ExecuteInternal() > @ 0x1860b12 impala::ImpalaServer::Execute() > @ 0x18cd0b6 impala::ImpalaServer::query() > @ 0x1df8ca0 beeswax::BeeswaxServiceProcessor::process_query() > @ 0x1df89ee beeswax::BeeswaxServiceProcessor::dispatchCall() > @ 0x1de11e9 impala::ImpalaServiceProcessor::dispatchCall() > @ 0x153fb5c apache::thrift::TDispatchProcessor::process() > @ 0x170eecd > apache::thrift::server::TAcceptQueueServer::Task::run() > @ 0x170ada7 impala::ThriftThread::RunRunnable() > @ 0x170c4b3 boost::_mfi::mf2<>::operator()() > @ 0x170c349 boost::_bi::list3<>::operator()<>() > @ 0x170c095 boost::_bi::bind_t<>::operator()() > @ 0x170bfa8 > boost::detail::function::void_function_obj_invoker0<>::invoke() > @ 0x171a89e boost::function0<>::operator()() > @ 0x19f289b impala::Thread::SuperviseThread() > @ 0x19fb42e boost::_bi::list4<>::operator()<>() > @ 0x19fb371 boost::_bi::bind_t<>::operator()() > @ 0x19fb334 boost::detail::thread_data<>::run() > @ 0x20a698a thread_proxy > @ 0x7f33eeb496ba start_thread > @ 0x7f33ee87f3dd clone -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (IMPALA-6332) Impala webserver should return HTTP error code for missing query profiles
Zoram Thanga created IMPALA-6332:
Summary: Impala webserver should return HTTP error code for
missing query profiles
Key: IMPALA-6332
URL: https://issues.apache.org/jira/browse/IMPALA-6332
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.10.0, Impala 2.11.0
Reporter: Zoram Thanga
Priority: Minor
Fix For: Impala 2.11.0
When we try to read the thrift runtime profile of a query soon after the query
is finished, sometimes the profile cannot be found in either the query map or
query log. Then the server sends back "Query id $0 not found." from
ImpalaServer::GetRuntimeProfileStr(). This is followed up in
ImpalaHttpHandler::QueryProfileEncodedHandler() as:
ss.str(Substitute("Could not obtain runtime profile: $0", status.GetDetail()));
The string us returned to the caller, but the HTTP response code is OK. This
can fool clients into thinking that they successfully read a valid thrift
profile. But since that's not true, clients that deserialize the thrift profile
may think that they received a corrupted profile.
We should change the code to send back a non-OK response, such as 404 in this
situation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (IMPALA-6307) A CTAS query fails with error: AnalysisException: Duplicate column name:
Zoram Thanga created IMPALA-6307:
Summary: A CTAS query fails with error: AnalysisException:
Duplicate column name:
Key: IMPALA-6307
URL: https://issues.apache.org/jira/browse/IMPALA-6307
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 2.10.0
Reporter: Zoram Thanga
Assignee: Zoram Thanga
The following query triggers the exception:
CREATE TABLE foo partitioned by (year) AS
WITH TMP AS (
SELECT a.timestamp_col, a.year FROM functional.alltypes a
LEFT JOIN functional.alltypes b
ON b.timestamp_col BETWEEN a.timestamp_col AND a.timestamp_col
)
SELECT a.timestamp_col, a.year FROM TMP a;
The exception is thrown from TableDef::analyzeColumnDefs():
private void analyzeColumnDefs(Analyzer analyzer) throws AnalysisException {
Set colNames = Sets.newHashSet();
for (ColumnDef colDef: columnDefs_) {
colDef.analyze(analyzer);
if (!colNames.add(colDef.getColName().toLowerCase())) {
throw new AnalysisException("Duplicate column name: " +
colDef.getColName());
}
if (!isKuduTable() && colDef.hasKuduOptions()) {
throw new AnalysisException(String.format("Unsupported column options
for " +
"file format '%s': '%s'", getFileFormat().name(),
colDef.toString()));
}
}
for (ColumnDef colDef: getPartitionColumnDefs()) {
colDef.analyze(analyzer);
if (!colDef.getType().supportsTablePartitioning()) {
throw new AnalysisException(
String.format("Type '%s' is not supported as partition-column type
" +
"in column: %s", colDef.getType().toSql(),
colDef.getColName()));
}
if (!colNames.add(colDef.getColName().toLowerCase())) {
* throw new AnalysisException("Duplicate column name: " +
colDef.getColName());*
}
}
The column duplication happens for "year" because it's in both columnDefs_ and
dataLayout_::partitionColDefs_.
The issue does not reproduce is we replace BETWEEN in the JOIN clause with the
equivalent "b.timestamp_col > a.timestamp_col AND b.timestamp_col <
a.timestamp_col".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (IMPALA-6271) Impala daemon should log a message when it's being shut down
Zoram Thanga created IMPALA-6271: Summary: Impala daemon should log a message when it's being shut down Key: IMPALA-6271 URL: https://issues.apache.org/jira/browse/IMPALA-6271 Project: IMPALA Issue Type: Improvement Components: Backend Affects Versions: Impala 2.10.0 Reporter: Zoram Thanga At present the Impala daemon does not log any message when it is being shut down, usually via SIGTERM from management software or OS shutdown. It would be good to at the very least catch this signal to log a message that we are going down. This will aid in serviceability. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (IMPALA-6225) IMPALA-5599 broke a client software due to precision change in date-time string
Zoram Thanga created IMPALA-6225: Summary: IMPALA-5599 broke a client software due to precision change in date-time string Key: IMPALA-6225 URL: https://issues.apache.org/jira/browse/IMPALA-6225 Project: IMPALA Issue Type: Bug Components: Backend Affects Versions: Impala 2.11.0 Reporter: Zoram Thanga Assignee: Zoram Thanga IMPALA-5599 changed the precision of Start Time and End Time date-time strings of query profiles to microseconds. This ended up breaking compatibility with a management client software. This JIRA is being raised to restore the precision to what it was (nanoseconds) , to address the breakage. Longer term, the interface contract between Impala and such client software will be made stricter and more efficient, and avoid having clients parse what are meant to be human-readable information. The Impala testing also needs to be enhanced to ensure interface contracts are not broken. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
