[jira] [Commented] (IMPALA-8316) Update re2 to avoid lock contention
[ https://issues.apache.org/jira/browse/IMPALA-8316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794146#comment-16794146 ] Todd Lipcon commented on IMPALA-8316: - I tried updating to the latest re2 and still saw lock contention on different locks. Got a reasonable (30-40%) speedup but I think IMPALA-5393 has the right fix. > Update re2 to avoid lock contention > --- > > Key: IMPALA-8316 > URL: https://issues.apache.org/jira/browse/IMPALA-8316 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Labels: perf > > I ran the following test query and found that it spent a lot of time in lock > contention within the re2 library: > ```select sum(l_linenumber) from item_20x where > regexp_extract(l_shipinstruct, '.*E', 0) like '%E' ;``` > I think this lock contention would happen on any regex that involves > backtracking. This was fixed in the re2 library upstream in > https://github.com/google/re2/commit/eb00dfdd82015be22086cacc6bf830f72a10e2bc#diff-a60a8d25ed15adf68b94c85775fd3cf7 > We should consider upgrading re2 to the latest release, or if not that, at > least cherry-picking this perf fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8316) Update re2 to avoid lock contention
[ https://issues.apache.org/jira/browse/IMPALA-8316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reassigned IMPALA-8316: --- Assignee: Todd Lipcon > Update re2 to avoid lock contention > --- > > Key: IMPALA-8316 > URL: https://issues.apache.org/jira/browse/IMPALA-8316 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Labels: perf > > I ran the following test query and found that it spent a lot of time in lock > contention within the re2 library: > ```select sum(l_linenumber) from item_20x where > regexp_extract(l_shipinstruct, '.*E', 0) like '%E' ;``` > I think this lock contention would happen on any regex that involves > backtracking. This was fixed in the re2 library upstream in > https://github.com/google/re2/commit/eb00dfdd82015be22086cacc6bf830f72a10e2bc#diff-a60a8d25ed15adf68b94c85775fd3cf7 > We should consider upgrading re2 to the latest release, or if not that, at > least cherry-picking this perf fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8316) Update re2 to avoid lock contention
[ https://issues.apache.org/jira/browse/IMPALA-8316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794129#comment-16794129 ] Todd Lipcon commented on IMPALA-8316: - (to be specific, the regexp_extract(l_shipinstruct, '.*E', 0) bit is the slow expression in this example, not the LIKE) > Update re2 to avoid lock contention > --- > > Key: IMPALA-8316 > URL: https://issues.apache.org/jira/browse/IMPALA-8316 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Todd Lipcon >Priority: Major > Labels: perf > > I ran the following test query and found that it spent a lot of time in lock > contention within the re2 library: > ```select sum(l_linenumber) from item_20x where > regexp_extract(l_shipinstruct, '.*E', 0) like '%E' ;``` > I think this lock contention would happen on any regex that involves > backtracking. This was fixed in the re2 library upstream in > https://github.com/google/re2/commit/eb00dfdd82015be22086cacc6bf830f72a10e2bc#diff-a60a8d25ed15adf68b94c85775fd3cf7 > We should consider upgrading re2 to the latest release, or if not that, at > least cherry-picking this perf fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8286) Accounting error in stress test on client failure
[ https://issues.apache.org/jira/browse/IMPALA-8286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794131#comment-16794131 ] ASF subversion and git services commented on IMPALA-8286: - Commit e73a007ee8209ac63e66e126d8488c11e3697f18 in impala's branch refs/heads/master from Tim Armstrong [ https://gitbox.apache.org/repos/asf?p=impala.git;h=e73a007 ] IMPALA-8286: fix accounting error on stress client exit Testing: Ran some cluster stress tests with this change. Change-Id: I389eb3a439f3c33dd192814cb80c9ca2bd8c5790 Reviewed-on: http://gerrit.cloudera.org:8080/12675 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Accounting error in stress test on client failure > - > > Key: IMPALA-8286 > URL: https://issues.apache.org/jira/browse/IMPALA-8286 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.2.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > > The change for IMPALA-6662 omitted incrementing > NUM_QUERIES_STARTED_RUNNING_OR_CANCELLED when a runner process exits, which > means that the accounting can be wrong when client processes fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8316) Update re2 to avoid lock contention
Todd Lipcon created IMPALA-8316: --- Summary: Update re2 to avoid lock contention Key: IMPALA-8316 URL: https://issues.apache.org/jira/browse/IMPALA-8316 Project: IMPALA Issue Type: Improvement Components: Backend Reporter: Todd Lipcon I ran the following test query and found that it spent a lot of time in lock contention within the re2 library: ```select sum(l_linenumber) from item_20x where regexp_extract(l_shipinstruct, '.*E', 0) like '%E' ;``` I think this lock contention would happen on any regex that involves backtracking. This was fixed in the re2 library upstream in https://github.com/google/re2/commit/eb00dfdd82015be22086cacc6bf830f72a10e2bc#diff-a60a8d25ed15adf68b94c85775fd3cf7 We should consider upgrading re2 to the latest release, or if not that, at least cherry-picking this perf fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8310) Terminate impalad when HDFS call thread pool is depleted
[ https://issues.apache.org/jira/browse/IMPALA-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794014#comment-16794014 ] Zoram Thanga commented on IMPALA-8310: -- Ahh yes. Thanks [~joemcdonnell]. > Terminate impalad when HDFS call thread pool is depleted > > > Key: IMPALA-8310 > URL: https://issues.apache.org/jira/browse/IMPALA-8310 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.2.0 >Reporter: Lars Volker >Priority: Major > > JVM hangs will currently consume the thread pool added in IMPALA-7738. Once > the pool is depleted we abort new queries. Instead we should consider killing > the Impalad since it is in a state where it won’t be able to process queries > until it is restarted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8314) bootstrap_system.sh does not set correct permissions for files in ~/.ssh
[
https://issues.apache.org/jira/browse/IMPALA-8314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hector Acosta updated IMPALA-8314:
--
Description:
bootstrap_system.sh may generate (or append to) ~/.ssh/authorized_keys and
~/.ssh/config which may result in the following error:
{code:java}
ssh localhost whoami
Bad owner or permissions on $HOME/.ssh/config
{code}
The correct permissions here are 0600 regardless of the umask setting.
was:
bootstrap_system.sh may generate (or append to) ~/.ssh/authorized_keys and
~/.ssh/config which may result in the following error:
{code:java}
ssh localhost whoami
Bad owner or permissions on $HOME/.ssh/config
{code}
> bootstrap_system.sh does not set correct permissions for files in ~/.ssh
>
>
> Key: IMPALA-8314
> URL: https://issues.apache.org/jira/browse/IMPALA-8314
> Project: IMPALA
> Issue Type: Bug
>Reporter: Hector Acosta
>Priority: Major
>
> bootstrap_system.sh may generate (or append to) ~/.ssh/authorized_keys and
> ~/.ssh/config which may result in the following error:
> {code:java}
> ssh localhost whoami
> Bad owner or permissions on $HOME/.ssh/config
> {code}
> The correct permissions here are 0600 regardless of the umask setting.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8314) bootstrap_system.sh does not set correct permissions for files in ~/.ssh
[
https://issues.apache.org/jira/browse/IMPALA-8314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hector Acosta updated IMPALA-8314:
--
Description:
bootstrap_system.sh may generate (or append to) ~/.ssh/authorized_keys and
~/.ssh/config which may result in the following error:
{code:java}
ssh localhost whoami
Bad owner or permissions on $HOME/.ssh/config
{code}
> bootstrap_system.sh does not set correct permissions for files in ~/.ssh
>
>
> Key: IMPALA-8314
> URL: https://issues.apache.org/jira/browse/IMPALA-8314
> Project: IMPALA
> Issue Type: Bug
>Reporter: Hector Acosta
>Priority: Major
>
> bootstrap_system.sh may generate (or append to) ~/.ssh/authorized_keys and
> ~/.ssh/config which may result in the following error:
>
>
> {code:java}
> ssh localhost whoami
> Bad owner or permissions on $HOME/.ssh/config
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8305) Generate JUnitXML symptom to detect crashes due to a DCHECK
[
https://issues.apache.org/jira/browse/IMPALA-8305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell reassigned IMPALA-8305:
-
Assignee: Joe McDonnell
> Generate JUnitXML symptom to detect crashes due to a DCHECK
> ---
>
> Key: IMPALA-8305
> URL: https://issues.apache.org/jira/browse/IMPALA-8305
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 3.3.0
>Reporter: Joe McDonnell
>Assignee: Joe McDonnell
>Priority: Major
>
> When an impalad or other process hits a DCHECK, it outputs a FATAL log with
> information about the check that failed. For example:
> {noformat}
> Log file created at: 2019/02/17 00:27:43
> Running on machine:
> impala-ec2-centos74-r4-4xlarge-ondemand-0a7e.vpc.cloudera.com
> Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg
> F0217 00:27:43.730983 109290 query-state.cc:604]
> 154c7230f098d1a5:e75d0b4] Check failed: is_cancelled_.Load() == 1 (0
> vs. 1){noformat}
> Like all impalad crashes, this can show up as random tests failing. This
> FATAL log should generate a JUnitXML symptom containing the log output.
> The check will need to navigate some exceptions. For example, some backend
> tests intentionally hit DCHECKs (e.g. buffer-pool-test and some others).
> Also, there is a custom cluster test that fails impalad startup and generates
> a FATAL log that is not a DCHECK.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8310) Terminate impalad when HDFS call thread pool is depleted
[ https://issues.apache.org/jira/browse/IMPALA-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793982#comment-16793982 ] Joe McDonnell commented on IMPALA-8310: --- I think this would be the right thing to do. We should dump appropriate state when it happens. Another option is to report this node as incapable of executing queries to the statestore, but I think I prefer aborting the impalad. [~zoram] The caller thread messages a thread pool, which does the actual IO. When the caller times out, the thread doing the actual IO call is still stuck. The thread pool has a fixed capacity and can run out. > Terminate impalad when HDFS call thread pool is depleted > > > Key: IMPALA-8310 > URL: https://issues.apache.org/jira/browse/IMPALA-8310 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.2.0 >Reporter: Lars Volker >Priority: Major > > JVM hangs will currently consume the thread pool added in IMPALA-7738. Once > the pool is depleted we abort new queries. Instead we should consider killing > the Impalad since it is in a state where it won’t be able to process queries > until it is restarted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8315) Error from shutil.rmtree in ImpalaTestSuite::run_stmt_in_hive()
Joe McDonnell created IMPALA-8315:
-
Summary: Error from shutil.rmtree in
ImpalaTestSuite::run_stmt_in_hive()
Key: IMPALA-8315
URL: https://issues.apache.org/jira/browse/IMPALA-8315
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.3.0
Reporter: Joe McDonnell
TestHBaseQueries.test_hbase_col_filter() failed on a centos6 exhaustive build
with the following error:
{noformat}
query_test/test_hbase_queries.py:89: in test_hbase_col_filter
self.run_stmt_in_hive(add_data)
common/impala_test_suite.py:781: in run_stmt_in_hive
if tmpdir is not None: shutil.rmtree(tmpdir)
/usr/lib64/python2.6/shutil.py:212: in rmtree
rmtree(fullname, ignore_errors, onerror)
/usr/lib64/python2.6/shutil.py:212: in rmtree
rmtree(fullname, ignore_errors, onerror)
/usr/lib64/python2.6/shutil.py:212: in rmtree
rmtree(fullname, ignore_errors, onerror)
/usr/lib64/python2.6/shutil.py:217: in rmtree
onerror(os.remove, fullname, sys.exc_info())
/usr/lib64/python2.6/shutil.py:215: in rmtree
os.remove(fullname)
E OSError: [Errno 2] No such file or directory:
'/tmp/impala-tests-j2b9bQ/localRunner/jenkins/job_local541814598_0023/job_local541814598_0023.xml'{noformat}
ImpalaTestSuite::run_stmt_in_hive() creates a temporary directory when in
LocalRunner mode to avoid different tests conflicting on the directory name. It
creates the temp directory, runs the Hive statement, then removes the directory
at the end. The removal is failing here. It looks like there might be a race
between pytest running shutil.rmtree() and Hive cleaning up its own files from
the statement.
We might want to use shutil.rmtree(..., ignore_errors=True).
Seen once.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8314) bootstrap_system.sh does not set correct permissions for files in ~/.ssh
[ https://issues.apache.org/jira/browse/IMPALA-8314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793964#comment-16793964 ] Jim Apple commented on IMPALA-8314: --- Hi Hector! Thank you for filing this ticket. Can you please add some notes to the Description section explaining exactly what you mean: which files, what permissions are set, what permissions should be set, and so on? > bootstrap_system.sh does not set correct permissions for files in ~/.ssh > > > Key: IMPALA-8314 > URL: https://issues.apache.org/jira/browse/IMPALA-8314 > Project: IMPALA > Issue Type: Bug >Reporter: Hector Acosta >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-337) Update Impala debug webserver to accept HTTP POST requests
[ https://issues.apache.org/jira/browse/IMPALA-337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell resolved IMPALA-337. -- Resolution: Fixed Fix Version/s: Impala 3.3.0 > Update Impala debug webserver to accept HTTP POST requests > -- > > Key: IMPALA-337 > URL: https://issues.apache.org/jira/browse/IMPALA-337 > Project: IMPALA > Issue Type: Task > Components: Distributed Exec >Affects Versions: Impala 1.1 >Reporter: Lenni Kuff >Assignee: Joe McDonnell >Priority: Minor > Fix For: Impala 3.3.0 > > > The Impala debug webserver doesn't currently handle HTTP POST requests. It > appears the requested handler is being called, but the request is being > treated as a GET. It would be useful to update the webserver to accept both > GET and POST commands. > This is something that is required if we want to have Impala to provide > automatic symbol resolution for remote profiling of the process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-337) Update Impala debug webserver to accept HTTP POST requests
[ https://issues.apache.org/jira/browse/IMPALA-337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell updated IMPALA-337: - Target Version: Impala 3.3.0 (was: Product Backlog) > Update Impala debug webserver to accept HTTP POST requests > -- > > Key: IMPALA-337 > URL: https://issues.apache.org/jira/browse/IMPALA-337 > Project: IMPALA > Issue Type: Task > Components: Distributed Exec >Affects Versions: Impala 1.1 >Reporter: Lenni Kuff >Assignee: Joe McDonnell >Priority: Minor > > The Impala debug webserver doesn't currently handle HTTP POST requests. It > appears the requested handler is being called, but the request is being > treated as a GET. It would be useful to update the webserver to accept both > GET and POST commands. > This is something that is required if we want to have Impala to provide > automatic symbol resolution for remote profiling of the process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-337) Update Impala debug webserver to accept HTTP POST requests
[ https://issues.apache.org/jira/browse/IMPALA-337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell reassigned IMPALA-337: Assignee: Joe McDonnell > Update Impala debug webserver to accept HTTP POST requests > -- > > Key: IMPALA-337 > URL: https://issues.apache.org/jira/browse/IMPALA-337 > Project: IMPALA > Issue Type: Task > Components: Distributed Exec >Affects Versions: Impala 1.1 >Reporter: Lenni Kuff >Assignee: Joe McDonnell >Priority: Minor > > The Impala debug webserver doesn't currently handle HTTP POST requests. It > appears the requested handler is being called, but the request is being > treated as a GET. It would be useful to update the webserver to accept both > GET and POST commands. > This is something that is required if we want to have Impala to provide > automatic symbol resolution for remote profiling of the process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-337) Update Impala debug webserver to accept HTTP POST requests
[ https://issues.apache.org/jira/browse/IMPALA-337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-337 started by Joe McDonnell. > Update Impala debug webserver to accept HTTP POST requests > -- > > Key: IMPALA-337 > URL: https://issues.apache.org/jira/browse/IMPALA-337 > Project: IMPALA > Issue Type: Task > Components: Distributed Exec >Affects Versions: Impala 1.1 >Reporter: Lenni Kuff >Assignee: Joe McDonnell >Priority: Minor > > The Impala debug webserver doesn't currently handle HTTP POST requests. It > appears the requested handler is being called, but the request is being > treated as a GET. It would be useful to update the webserver to accept both > GET and POST commands. > This is something that is required if we want to have Impala to provide > automatic symbol resolution for remote profiling of the process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8314) bootstrap_system.sh does not set correct permissions for files in ~/.ssh
Hector Acosta created IMPALA-8314: - Summary: bootstrap_system.sh does not set correct permissions for files in ~/.ssh Key: IMPALA-8314 URL: https://issues.apache.org/jira/browse/IMPALA-8314 Project: IMPALA Issue Type: Bug Reporter: Hector Acosta -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8277) CHECK can be hit when there are gaps in present CPU numbers (KUDU-2721)
[ https://issues.apache.org/jira/browse/IMPALA-8277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-8277. --- Resolution: Fixed Fix Version/s: Impala 3.3.0 > CHECK can be hit when there are gaps in present CPU numbers (KUDU-2721) > --- > > Key: IMPALA-8277 > URL: https://issues.apache.org/jira/browse/IMPALA-8277 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.1.0, Impala 3.2.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Critical > Labels: crash > Fix For: Impala 3.3.0 > > > This is a placeholder to port KUDU-2721 to our gutil once it's fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8299) GroupingAggregator::Partition::Close() may access an uninitialized hash table
[
https://issues.apache.org/jira/browse/IMPALA-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Tauber-Marshall resolved IMPALA-8299.
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> GroupingAggregator::Partition::Close() may access an uninitialized hash table
> -
>
> Key: IMPALA-8299
> URL: https://issues.apache.org/jira/browse/IMPALA-8299
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0, Impala 3.2.0
>Reporter: Michael Ho
>Assignee: Thomas Tauber-Marshall
>Priority: Blocker
> Labels: crash
> Fix For: Impala 3.3.0
>
>
> On the rare occasion that {{Suballocator::Allocate()}} failed in
> {{HashTable::init()}}, the {{GroupingAggregator::Partition::Close()}} may
> access an uninitialized hash table, leading to crash below:
> {noformat}
> #4 0x7f5413a1268f in JVM_handle_linux_signal () from
> ./sysroot/usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #5 0x7f5413a08be3 in signalHandler(int, siginfo*, void*) () from
> ./sysroot/usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x023c5c00 in impala::HashTable::NextFilledBucket
> (this=0x1a1fa000, bucket_idx=0x7f533cfc73d0, node=0x7f533cfc73c8)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hash-table.inline.h:185
> #8 0x0244b639 in impala::HashTable::Begin (this=0x1a1fa000,
> ctx=0x15445e00) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hash-table.inline.h:163
> #9 0x02457c69 in impala::GroupingAggregator::Partition::Close
> (this=0x133ef3e0, finalize_rows=true)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/grouping-aggregator-partition.cc:207
> #10 0x02448f26 in impala::GroupingAggregator::ClosePartitions
> (this=0x1327aa00) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/grouping-aggregator.cc:939
> #11 0x02443622 in impala::GroupingAggregator::Close (this=0x1327aa00,
> state=0x1bf5f180) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/grouping-aggregator.cc:386
> #12 0x02412ce4 in impala::AggregationNode::Close (this=0x1346f600,
> state=0x1bf5f180) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/aggregation-node.cc:139
> #13 0x0242a69f in impala::BlockingJoinNode::ProcessBuildInputAsync
> (this=0xb466480, state=0x1bf5f180, build_sink=0xf35f600,
> status=0x7f53402ddb20)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/blocking-join-node.cc:173
> #14 0x0242a865 in
> impala::BlockingJoinNodeoperator()(void) const
> (__closure=0x21c6fd80) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/blocking-join-node.cc:212
> #15 0x0242c4d5 in
> boost::detail::function::void_function_obj_invoker0 impala::DataSink*)::,
> void>::invoke(boost::detail::function::function_buffer &)
> (function_obj_ptr=...) at
> /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #16 0x01d7eb4e in boost::function0::operator()
> (this=0x7f533cfc7ba0) at
> /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #17 0x0224f3d1 in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*) (name=...,
> category=..., functor=..., parent_thread_info=0x7f53402de850,
> thread_started=0x7f53402dd7d0) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:359
> #18 0x02257755 in boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value to continue, or q to
> quit---
> romise*> >::operator() const&, std::string const&, boost::function, impala::ThreadDebugInfo
> const*, impala::Promise*),
> boost::_bi::list0>(boost::_bi::type, void (*&)(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*), boost::_bi::list0&, int) (
> this=0x14cf5fc0,
> f=@0x14cf5fb8: 0x224f06a const&, std::string const&, boost::function, impala::ThreadDebugInfo
> const*, impala::Promise*)>, a=...)
> at
> /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525
> (gdb) f 7
> #7 0x023c5c00 in impala::HashTable::NextFilledBucket
> (this=0x1a1fa000,
[jira] [Commented] (IMPALA-8299) GroupingAggregator::Partition::Close() may access an uninitialized hash table
[
https://issues.apache.org/jira/browse/IMPALA-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793874#comment-16793874
]
ASF subversion and git services commented on IMPALA-8299:
-
Commit dee911ee6cc749b4c8172f0e644069e16bf5ef57 in impala's branch
refs/heads/master from Thomas Tauber-Marshall
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=dee911e ]
IMPALA-8299: Fix crash in GroupingAggregator on uninited hash table
If HashTable::Init() fails, in some cases we may try to call
HashTable::Close() on the table in GroupingAggregator::Close(), which
can cause a crash.
The solution is to set the hash table to a nullptr if Init() fails.
I also verified that the other use of HashTable, PhjBuilder, handles
this case correctly.
Testing:
- Manually verified that the crash no longer occurs if HashTable::Init
fails.
Change-Id: I381657c43f93eb4871b93d54532cb886198fd0b2
Reviewed-on: http://gerrit.cloudera.org:8080/12746
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> GroupingAggregator::Partition::Close() may access an uninitialized hash table
> -
>
> Key: IMPALA-8299
> URL: https://issues.apache.org/jira/browse/IMPALA-8299
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0, Impala 3.2.0
>Reporter: Michael Ho
>Assignee: Thomas Tauber-Marshall
>Priority: Blocker
> Labels: crash
>
> On the rare occasion that {{Suballocator::Allocate()}} failed in
> {{HashTable::init()}}, the {{GroupingAggregator::Partition::Close()}} may
> access an uninitialized hash table, leading to crash below:
> {noformat}
> #4 0x7f5413a1268f in JVM_handle_linux_signal () from
> ./sysroot/usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #5 0x7f5413a08be3 in signalHandler(int, siginfo*, void*) () from
> ./sysroot/usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x023c5c00 in impala::HashTable::NextFilledBucket
> (this=0x1a1fa000, bucket_idx=0x7f533cfc73d0, node=0x7f533cfc73c8)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hash-table.inline.h:185
> #8 0x0244b639 in impala::HashTable::Begin (this=0x1a1fa000,
> ctx=0x15445e00) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hash-table.inline.h:163
> #9 0x02457c69 in impala::GroupingAggregator::Partition::Close
> (this=0x133ef3e0, finalize_rows=true)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/grouping-aggregator-partition.cc:207
> #10 0x02448f26 in impala::GroupingAggregator::ClosePartitions
> (this=0x1327aa00) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/grouping-aggregator.cc:939
> #11 0x02443622 in impala::GroupingAggregator::Close (this=0x1327aa00,
> state=0x1bf5f180) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/grouping-aggregator.cc:386
> #12 0x02412ce4 in impala::AggregationNode::Close (this=0x1346f600,
> state=0x1bf5f180) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/aggregation-node.cc:139
> #13 0x0242a69f in impala::BlockingJoinNode::ProcessBuildInputAsync
> (this=0xb466480, state=0x1bf5f180, build_sink=0xf35f600,
> status=0x7f53402ddb20)
> at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/blocking-join-node.cc:173
> #14 0x0242a865 in
> impala::BlockingJoinNodeoperator()(void) const
> (__closure=0x21c6fd80) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/blocking-join-node.cc:212
> #15 0x0242c4d5 in
> boost::detail::function::void_function_obj_invoker0 impala::DataSink*)::,
> void>::invoke(boost::detail::function::function_buffer &)
> (function_obj_ptr=...) at
> /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #16 0x01d7eb4e in boost::function0::operator()
> (this=0x7f533cfc7ba0) at
> /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #17 0x0224f3d1 in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*) (name=...,
> category=..., functor=..., parent_thread_info=0x7f53402de850,
> thread_started=0x7f53402dd7d0) at
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:359
> #18 0x02257755 in boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
>
[jira] [Commented] (IMPALA-988) Join strategy (broadcast vs shuffle) decision does not take mem limit and other joins into account
[ https://issues.apache.org/jira/browse/IMPALA-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793875#comment-16793875 ] ASF subversion and git services commented on IMPALA-988: Commit 4b83ff1a8d0cf97d45d01a89456f48a57031291d in impala's branch refs/heads/master from Alex Rodoni [ https://gitbox.apache.org/repos/asf?p=impala.git;h=4b83ff1 ] [DOCS] Removed IMPALA-988 from the Known Issues list Change-Id: I93e8ca3d0d070a75bb610f6f94c8bd78ddd5024f Reviewed-on: http://gerrit.cloudera.org:8080/12767 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins > Join strategy (broadcast vs shuffle) decision does not take mem limit and > other joins into account > -- > > Key: IMPALA-988 > URL: https://issues.apache.org/jira/browse/IMPALA-988 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 1.2.1 >Reporter: Alan Choi >Priority: Minor > Labels: resource-management > > The amount of available memory changes the trade-off between partitioned and > shuffle join strategies: if switching to shuffle join can avoid spilling to > disk, it may be worth paying the cost of the additional network transfer. > There are two issues: > 1. Join strategy decision only takes query mem-limit into account but ignore > process mem-limit. > 2. Join strategy decision does not take other joins of the same query into > account. When multiple joins are present, it'll go over the mem-limit. > Note that when IMPALA-3200 is completed, this shouldn't prevent the query > running to completion, but still affects performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8277) CHECK can be hit when there are gaps in present CPU numbers (KUDU-2721)
[ https://issues.apache.org/jira/browse/IMPALA-8277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793873#comment-16793873 ] ASF subversion and git services commented on IMPALA-8277: - Commit b79a03bf408e32d0a29ead33c788b84e7bf1d439 in impala's branch refs/heads/master from Tim Armstrong [ https://gitbox.apache.org/repos/asf?p=impala.git;h=b79a03b ] IMPALA-8277: support ranges in CPU lists Generalise the logic to handle comma-separated ranges. Testing: Added unit test for parsing logic to test the various possible permutations of CPU ranges. Porting notes: This is a port of KUDU-2721. There were some minor conflicts around include ordering in sysinfo.cc that I resolved by switching more of includes to match Kudu's version. I also made sure to make sysinfo-test runnable without modifications to the source file, which is possible with the new unified backend test infrastructure. Change-Id: I97311bfbcf70bea069e941b6e7f4f015fb781b3f Reviewed-on: http://gerrit.cloudera.org:8080/12657 Reviewed-by: Adar Dembo Tested-by: Kudu Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/12755 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > CHECK can be hit when there are gaps in present CPU numbers (KUDU-2721) > --- > > Key: IMPALA-8277 > URL: https://issues.apache.org/jira/browse/IMPALA-8277 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.1.0, Impala 3.2.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Critical > Labels: crash > > This is a placeholder to port KUDU-2721 to our gutil once it's fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-3189) Address scalability issue with N^2 KDC requests on cluster startup
[
https://issues.apache.org/jira/browse/IMPALA-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho updated IMPALA-3189:
---
Priority: Critical (was: Major)
> Address scalability issue with N^2 KDC requests on cluster startup
> --
>
> Key: IMPALA-3189
> URL: https://issues.apache.org/jira/browse/IMPALA-3189
> Project: IMPALA
> Issue Type: Improvement
> Components: Distributed Exec, Security
>Affects Versions: Impala 2.5.0
>Reporter: Henry Robinson
>Priority: Critical
> Labels: kerberos, scalability
>
> When Impala runs a query that shuffles data amongst all nodes in a
> Kerberos-secured cluster, every node will need to acquire a TGS for every
> other node. In a cluster of 100 nodes or more, this can overwhelm the KDC,
> and queries can exit with an error ("Could not contact KDC for realm").
> A simple workaround is to run a warm-up query until it succeeds (which can
> take a few minutes after cluster startup). The KDC can also be scaled (e.g.
> with secondary KDC nodes).
> Impala can also consider either forcing a TGS request on start-up in a
> staggered fashion, or we can move to recommending SSL + client certificates
> for server<->server communication.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-4356) Automatically codegen expressions with any root Expr node
[ https://issues.apache.org/jira/browse/IMPALA-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793806#comment-16793806 ] Tim Armstrong commented on IMPALA-4356: --- Pooja handed off a WIP patch to me. Will see if I can revive it, although it needs some tweaks. > Automatically codegen expressions with any root Expr node > - > > Key: IMPALA-4356 > URL: https://issues.apache.org/jira/browse/IMPALA-4356 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.8.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: codegen > > Currently Impala only automatically codegens expression subtrees with > ScalarFnCall at the root. This is the expression type used to implement many > but not all expressions (including most builtin operators). One example of an > expression that is not automatically codegened is "case" statements. > The crux of this is to move ScalarFnCall::scalar_fn_wrapper_ into ScalarExpr > (and probably rename it). There are some consequential changes required to > make this work. Instead of each ScalarExpr subclass overriding Get*Val(), I > think Get*Val() should be a non-virtual method of ScalarExpr that either > calls the codegen'd function or calls into Get*ValInterpreted(), which would > be the new virtual function. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-4356) Automatically codegen expressions with any root Expr node
[ https://issues.apache.org/jira/browse/IMPALA-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong reassigned IMPALA-4356: - Assignee: Tim Armstrong (was: Pooja Nilangekar) > Automatically codegen expressions with any root Expr node > - > > Key: IMPALA-4356 > URL: https://issues.apache.org/jira/browse/IMPALA-4356 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.8.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: codegen > > Currently Impala only automatically codegens expression subtrees with > ScalarFnCall at the root. This is the expression type used to implement many > but not all expressions (including most builtin operators). One example of an > expression that is not automatically codegened is "case" statements. > The crux of this is to move ScalarFnCall::scalar_fn_wrapper_ into ScalarExpr > (and probably rename it). There are some consequential changes required to > make this work. Instead of each ScalarExpr subclass overriding Get*Val(), I > think Get*Val() should be a non-virtual method of ScalarExpr that either > calls the codegen'd function or calls into Get*ValInterpreted(), which would > be the new virtual function. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
