[jira] [Resolved] (IMPALA-7177) Impalad hangs when starting up
[
https://issues.apache.org/jira/browse/IMPALA-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-7177.
-
Resolution: Duplicate
> Impalad hangs when starting up
> --
>
> Key: IMPALA-7177
> URL: https://issues.apache.org/jira/browse/IMPALA-7177
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Lars Volker
>Priority: Critical
> Labels: broken-build
>
> In
> custom_cluster/test_breakpad.py::TestBreakpadExhaustive::test_minidump_cleanup_thread,
> a impalad hangs for 4 minutes when starting up. The tail of its log is:
> {noformat}
> I0613 14:55:55.514082 23644 init.cc:246] OS version: Linux version
> 2.6.32-358.14.1.el6.centos.plus.x86_64 (mockbu...@c6b9.bsys.dev.centos.org)
> (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Tue Jul 16
> 21:33:24 UTC 2013
> Clock: clocksource: 'xen', clockid_t: CLOCK_MONOTONIC_COARSE
> I0613 14:55:55.514091 23644 init.cc:247] Process ID: 23644
> I0613 14:55:55.514098 23644 init.cc:248] Default AES cipher mode for
> spill-to-disk: AES-CTR
> I0613 14:55:58.153287 23644 llvm-codegen.cc:152] CPU class for runtime code
> generation: sandybridge
> I0613 14:55:58.153379 23644 llvm-codegen.cc:154] Detected CPU flags:
> +sse2,+cx16,-tbm,-avx512ifma,-avx512dq,-fma4,-prfchw,-bmi2,-xsavec,-fsgsbase,+popcnt,-aes,-xsaves,-avx512er,-avx512vpopcntdq,-clwb,-avx512f,-clzero,-pku,+mmx,-lwp$
> I0613 14:55:58.153432 23644 llvm-codegen.cc:157] CPU flags enabled for
> runtime code generation:
> +sse2,+cx16,-tbm,-avx512ifma,-avx512dq,-fma4,-prfchw,-bmi2,-xsavec,-fsgsbase,+popcnt,-aes,-xsaves,-avx512er,-avx512vpopcntdq,-clwb,-avx$
> I0613 14:56:00.633289 23644 GlogAppender.java:137] Logging (re)initialized.
> Impala: VLOG, All other: INFO
> I0613 14:56:00.638018 23644 JniFrontend.java:139] Authorization is 'DISABLED'.
> I0613 14:56:00.640646 23644 JniFrontend.java:141] Java Input arguments:
> -agentlib:jdwp=transport=dt_socket,address=30002,server=y,suspend=n
> -Djava.library.path=/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/Impala-Toolchain/cdh_components/hadoop-3.0.0-cdh6.x-417234//lib/native/
> -XX:ErrorFile=$
> Java System properties:
> awt.toolkit:sun.awt.X11.XToolkit
> file.encoding.pkg:sun.io
> java.specification.version:1.8
> sun.cpu.isalist:
> sun.jnu.encoding:UTF-8
> java.class.path:/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/fe/src/test/resources:/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/fe/target/classes:/data/jenkins/workspace/impala-cdh6$
> I0613 14:56:00.691942 23644 HiveConf.java:188] Found configuration file
> file:/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/fe/src/test/resources/hive-site.xml
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Reopened] (IMPALA-6642) start-impala-cluster.py failing in some custom cluster tests
[
https://issues.apache.org/jira/browse/IMPALA-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang reopened IMPALA-6642:
-
Saw it again. This time in test_minidump_cleanup_thread.
> start-impala-cluster.py failing in some custom cluster tests
>
>
> Key: IMPALA-6642
> URL: https://issues.apache.org/jira/browse/IMPALA-6642
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.12.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Taras Bobrovytsky
>Priority: Critical
> Labels: broken-build
>
> Seen in recent builds, both on the 2.x branch. Not quite the same test/error,
> but they seem similar enough to probably be related:
> {noformat}
> 17:01:18 _ ERROR at setup of
> TestAdmissionController.test_require_user __
> 17:01:18 common/custom_cluster_test_suite.py:109: in setup_method
> 17:01:18 self._start_impala_cluster(cluster_args)
> 17:01:18 common/custom_cluster_test_suite.py:144: in _start_impala_cluster
> 17:01:18 check_call(cmd + options, close_fds=True)
> 17:01:18 /usr/lib64/python2.6/subprocess.py:505: in check_call
> 17:01:18 raise CalledProcessError(retcode, cmd)
> 17:01:18 E CalledProcessError: Command
> '['/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/bin/start-impala-cluster.py',
> '--cluster_size=3', '--num_coordinators=3',
> '--log_dir=/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests',
> '--log_level=1', '--impalad_args="-vmodule admission-controller=3
> -fair_scheduler_allocation_path
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/fair-scheduler-test2.xml
> -llama_site_path
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/llama-site-test2.xml
> -disable_admission_control=false -require_username" ',
> '--state_store_args="-statestore_heartbeat_frequency_ms=100
> -statestore_priority_update_frequency_ms=100" ']' returned non-zero exit
> status 1
> 17:01:18 Captured stdout setup
> -
> 17:01:18 Starting State Store logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 17:01:18 Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 17:01:18 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 17:01:18 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 17:01:18 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 17:01:18 Error starting cluster: Expected 3 impalad(s), only 2 found
> 17:01:18
> 17:01:18 Captured stderr setup
> -
> 17:01:18 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 17:01:18 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> {noformat}
> {noformat}
> 16:42:41 ___ ERROR at setup of
> TestAuthorization.test_access_runtime_profile
> 16:42:41 common/custom_cluster_test_suite.py:109: in setup_method
> 16:42:41 self._start_impala_cluster(cluster_args)
> 16:42:41 common/custom_cluster_test_suite.py:144: in _start_impala_cluster
> 16:42:41 check_call(cmd + options, close_fds=True)
> 16:42:41 /usr/lib64/python2.6/subprocess.py:505: in check_call
> 16:42:41 raise CalledProcessError(retcode, cmd)
> 16:42:41 E CalledProcessError: Command
> '['/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/bin/start-impala-cluster.py',
> '--cluster_size=3', '--num_coordinators=3',
> '--log_dir=/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests',
> '--log_level=1', '--impalad_args="--server_name=server1
> --authorization_policy_file=/test-warehouse/authz-policy.ini
> --authorized_proxy_user_config=hue=jenkins" ']' returned non-zero exit status
> 1
> 16:42:41 Captured stdout setup
> -
> 16:42:41 Starting State Store logging to
> /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 16:42:41 Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 16:42:41 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad.INFO
>
[jira] [Assigned] (IMPALA-6918) Implement COMMENT ON COLUMN
[
https://issues.apache.org/jira/browse/IMPALA-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fredy Wijaya reassigned IMPALA-6918:
Assignee: Fredy Wijaya
> Implement COMMENT ON COLUMN
> ---
>
> Key: IMPALA-6918
> URL: https://issues.apache.org/jira/browse/IMPALA-6918
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
>Reporter: Fredy Wijaya
>Assignee: Fredy Wijaya
>Priority: Minor
>
> Syntax:
> {noformat}
> COMMENT ON COLUMN my_table.my_column IS 'Employee ID Number';{noformat}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-5605) configuration to prevent crashes caused by thread resource limits
[
https://issues.apache.org/jira/browse/IMPALA-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-5605:
--
Target Version: Product Backlog
> configuration to prevent crashes caused by thread resource limits
> -
>
> Key: IMPALA-5605
> URL: https://issues.apache.org/jira/browse/IMPALA-5605
> Project: IMPALA
> Issue Type: Documentation
> Components: Docs
>Affects Versions: Impala 2.9.0
>Reporter: Matthew Mulder
>Assignee: Tim Armstrong
>Priority: Major
> Fix For: Impala 2.10.0
>
>
> Depending on the workload, Impala may need to create a very large number of
> threads. If so, it is necessary to configure the system correctly to prevent
> Impala from crashing because of resource limitations. Such a crash would look
> like this:{code}F0629 08:20:02.956413 29088 llvm-codegen.cc:111] LLVM hit
> fatal error: Unable to allocate section memory!
> terminate called after throwing an instance of
> 'boost::exception_detail::clone_impl
> >'{code}To prevent this, each Impala host should be configured like
> this:{code}echo 200 > /proc/sys/kernel/threads-max
> echo 200 > /proc/sys/kernel/pid_max
> echo 800 > /proc/sys/vm/max_map_count{code}In /etc/security/limits.conf
> add{code}impala soft nproc 262144
> impala hard nproc 262144{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-6816) Statestore spends a lot of time in GetMinSubscriberTopicVersion()
[
https://issues.apache.org/jira/browse/IMPALA-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-6816:
--
Target Version: Product Backlog
> Statestore spends a lot of time in GetMinSubscriberTopicVersion()
> -
>
> Key: IMPALA-6816
> URL: https://issues.apache.org/jira/browse/IMPALA-6816
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Minor
> Labels: admission-control, statestore
>
> {noformat}
> Samples: 13K of event 'cycles', Event count (approx.): 1200870513
> 20.23% statestored impalad [.]
> impala::Statestore::GetMinSubscriberTopicVersion(std::string const&,
> std::string*)
>7.68% statestored [kernel.kallsyms][k] find_busiest_group
>3.46% statestored impalad [.]
> impala::Statestore::Subscriber::LastTopicVersionProcessed(std::string const&)
> const
>3.26% statestored libc-2.12.so [.] __memcmp_sse4_1
>1.41% statestored [kernel.kallsyms][k] find_next_bit
>1.40% statestored [kernel.kallsyms][k] cpumask_next_and
>1.21% statestored libpthread-2.12.so [.] pthread_mutex_lock
>1.04% statestored libc-2.12.so [.] memcpy
>1.01% statestored [kernel.kallsyms][k] _spin_lock
>0.98% statestored impalad [.] 0x0088f903
>0.93% statestored impalad [.] 0x0088f8f5
>0.91% statestored impalad [.] 0x0088f8ea
>0.85% statestored [kernel.kallsyms][k] ixgbe_xmit_frame_ring
>0.77% statestored impalad [.] 0x0088f8e3
>0.75% statestored impalad [.] 0x0088f900
>0.75% statestored impalad [.]
> impala::Statestore::IsPrioritizedTopic(std::string const&)
>0.73% statestored impalad [.] 0x0088f8fa
>0.72% statestored impalad [.] operator new[](unsigned
> long)
>0.68% statestored [kernel.kallsyms][k] tcp_recvmsg
>0.67% statestored impalad [.] 0x0088f8fd
>0.66% statestored impalad [.]
> impala::Statestore::Topic::BuildDelta(std::string const&, long,
> impala::TTopicDelta*)
>0.61% statestored [kernel.kallsyms][k] thread_return
>0.60% statestored impalad [.] 0x0088f8f2
>0.60% statestored libstdc++.so.6 [.]
> std::string::compare(std::string const&) const
>0.59% statestored impalad [.] 0x0088f8e6
>0.56% statestored impalad [.] 0x0088f8ee
>0.56% statestored libcrypto.so.1.0.1e [.] aesni_encrypt
>0.55% statestored impalad [.] 0x0088f8e0
>0.55% statestored [kernel.kallsyms][k] tcp_transmit_skb
>0.53% statestored [kernel.kallsyms][k] fget_light
>0.51% statestored impalad [.] std::_Rb_tree std::pair >,
> std::_Select1st0.50% statestored impalad [.]
> apache::thrift::transport::TVirtualTransport apache::thrift::transport::TBufferBase>::readAll_virt(unsigned char*
>0.50% statestored impalad [.]
> impala::Statestore::DoSubscriberUpdate(impala::Statestore::UpdateKind, int,
> impala::Statestore::ScheduledSubscriberUpdate const&)
>0.49% statestored libssl.so.1.0.1e [.] tls1_enc
>0.48% statestored libssl.so.1.0.1e [.] ssl3_read_bytes
> {noformat}
> We are spending most of our time computing this for non-catalog topics, where
> it's not even used.
> There are a couple of ways we could fix this that I can think of:
> * Avoid including this information for topics where we're not interested in it
> * Cache or precompute the value somehow to avoid iterating over all
> subscribers every time
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7174) TestAdmissionController.test_cancellation failed with incorrect total-admitted metric
[
https://issues.apache.org/jira/browse/IMPALA-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7174:
--
Target Version: Impala 3.1.0
> TestAdmissionController.test_cancellation failed with incorrect
> total-admitted metric
> -
>
> Key: IMPALA-7174
> URL: https://issues.apache.org/jira/browse/IMPALA-7174
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build
> Attachments:
> impalad.ec2-m2-4xlarge-centos-6-4-01f7.vpc.cloudera.com.jenkins.log.INFO.20180614-060607.2553
>
>
> The failed revision is ee9a9b6c5000cf915716a15ea8a0b3605290a9a5, an
> descendant of 'IMPALA-5216: Make admission control queuing async'.
> {noformat}
> Stacktrace
> custom_cluster/test_admission_controller.py:557: in test_cancellation
> assert self.cluster.impalads[0].service.get_metric_value(
> E assert 0 == 3
> E+ where 0 = 0x4ed1b90>>('admission-controller.total-admitted.default-pool')
> E+where > =
> 0x4ed1b90>.get_metric_value
> E+ where 0x4ed1b90> = 0x52474d0>.service
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6223) Gracefully handle malformed 'with' queries in impala-shell
[
https://issues.apache.org/jira/browse/IMPALA-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-6223:
-
Assignee: Pooja Nilangekar
> Gracefully handle malformed 'with' queries in impala-shell
> --
>
> Key: IMPALA-6223
> URL: https://issues.apache.org/jira/browse/IMPALA-6223
> Project: IMPALA
> Issue Type: Bug
> Components: Clients
>Affects Versions: Impala 2.10.0
>Reporter: bharath v
>Assignee: Pooja Nilangekar
>Priority: Minor
> Labels: newbie
>
> Impala shell can throw a lexer error if it encounters a malformed "with"
> query.
> {noformat}
> impala-shell.sh -q "with foo as (select bar from temp where temp.a='"
> Starting Impala Shell without Kerberos authentication
> Connected to localhost:21000
> Server version: impalad version 2.11.0-SNAPSHOT DEBUG (build
> 0ee1765f38082bc5c10aa37b23cb8e57caa57d4e)
> Traceback (most recent call last):
> File "/home/bharath/Impala/shell/impala_shell.py", line 1463, in
> execute_queries_non_interactive_mode(options, query_options)
> File "/home/bharath/Impala/shell/impala_shell.py", line 1338, in
> execute_queries_non_interactive_mode
> shell.execute_query_list(queries)):
> File "/home/bharath/Impala/shell/impala_shell.py", line 1218, in
> execute_query_list
> if self.onecmd(q) is CmdStatus.ERROR:
> File "/home/bharath/Impala/shell/impala_shell.py", line 505, in onecmd
> return cmd.Cmd.onecmd(self, line)
> File "/usr/lib/python2.7/cmd.py", line 221, in onecmd
> return func(arg)
> File "/home/bharath/Impala/shell/impala_shell.py", line 1024, in do_with
> tokens = list(lexer)
> File "/usr/lib/python2.7/shlex.py", line 269, in next
> token = self.get_token()
> File "/usr/lib/python2.7/shlex.py", line 96, in get_token
> raw = self.read_token()
> File "/usr/lib/python2.7/shlex.py", line 172, in read_token
> raise ValueError, "No closing quotation"
> ValueError: No closing quotation
> {noformat}
> This happens because we use shlex to parse the input query to determine if
> its a DML and it can throw if the input doesn't have balanced quotes.
> {noformat}
> def do_with(self, args):
> """Executes a query with a WITH clause, fetching all rows"""
> query = self.imp_client.create_beeswax_query("with %s" % args,
> self.set_query_options)
> # Set posix=True and add "'" to escaped quotes
> # to deal with escaped quotes in string literals
> lexer = shlex.shlex(query.query.lstrip(), posix=True)
> lexer.escapedquotes += "'"
> # Because the WITH clause may precede DML or SELECT queries,
> # just checking the first token is insufficient.
> is_dml = False
> tokens = list(lexer) <
> {noformat}
> A simple shlex repro of that is as follows,
> {noformat}
> >>> lexer = shlex.shlex("with foo as (select bar from temp where temp.a='",
> >>> posix=True);
> >>> list(lexer)
> Traceback (most recent call last):
> File "", line 1, in
> File "/usr/lib/python2.7/shlex.py", line 269, in next
> token = self.get_token()
> File "/usr/lib/python2.7/shlex.py", line 96, in get_token
> raw = self.read_token()
> File "/usr/lib/python2.7/shlex.py", line 172, in read_token
> raise ValueError, "No closing quotation"
> ValueError: No closing quotation
> {noformat}
> Fix: Either catch the exception and handle it gracefully or have a better way
> to figure out the query type, using a SQL parser (more involved).
> This query also repros it:
> {code}
> with v as (select 1)
> select foo('\\'), ('bar
> ;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-6917) Implement COMMENT ON TABLE/VIEW
[
https://issues.apache.org/jira/browse/IMPALA-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fredy Wijaya resolved IMPALA-6917.
--
Resolution: Fixed
> Implement COMMENT ON TABLE/VIEW
> ---
>
> Key: IMPALA-6917
> URL: https://issues.apache.org/jira/browse/IMPALA-6917
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
>Reporter: Fredy Wijaya
>Assignee: Fredy Wijaya
>Priority: Minor
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> Syntax:
> {noformat}
> COMMENT ON TABLE my_db.my_table IS 'Employee Information';
> COMMENT ON VIEW my_db.my_view IS 'Employee Information';{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7115) Set a default THREAD_RESERVATION_LIMIT value
[
https://issues.apache.org/jira/browse/IMPALA-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-7115 started by Tim Armstrong.
-
> Set a default THREAD_RESERVATION_LIMIT value
>
>
> Key: IMPALA-7115
> URL: https://issues.apache.org/jira/browse/IMPALA-7115
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Labels: resource-management
>
> As a follow on to IMPALA-6035, we should set a default value that actually
> will help protect again insanely complex queries.
> Motivating discussion is here:
> https://gerrit.cloudera.org/#/c/10365/9/common/thrift/ImpalaInternalService.thrift
> {quote}
> Tim Armstrong
> 1:11 PM
> Dan suggested setting a default here. I started doing some experiments to see
> what our current practical limits are.
> On stock Ubuntu 16.04 I start getting thread_resource_error at around 8000
> reserved threads. I'm not sure that the config reflects what people would use
> on production systems so continuing to investigate.
> Dan Hecht
> 1:31 PM
> We could also consider choosing a default dynamically based on the OS's
> setting, if that's necessary.
> Tim Armstrong
> 3:45 PM
> I increased some of the configs (I think I was limited by
> /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max == 12288) and now it
> got oom-killed at ~26000 threads.
> I think unfortunately there are a lot of different OS knobs that impact this
> and they seem to evolve over time, so it's probably not feasible with a
> reasonable amount of effort to get it working on all common Linux distros.
> I was thinking ~5000, since 1000-2000 plan nodes is the most I've seen for a
> query running successfully in production.
> Maybe I should do this in a follow-on change, since we probably also want to
> add a test query at or near this limit.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7111) ASAN heap-use-after-free in impala::HdfsPluginTextScanner::CheckPluginEnabled
[
https://issues.apache.org/jira/browse/IMPALA-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-7111 started by Tim Armstrong.
-
> ASAN heap-use-after-free in impala::HdfsPluginTextScanner::CheckPluginEnabled
> -
>
> Key: IMPALA-7111
> URL: https://issues.apache.org/jira/browse/IMPALA-7111
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: asan, broken-build
>
> [~tarmstr...@cloudera.com] - I'm assigning this to you since you added this
> file in IMPALA-6941.
> {noformat}
> ==4582==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x603000e8aa28 at pc 0x017ab9b4 bp 0x7f67e5f6b650 sp 0x7f67e5f6b648
> READ of size 1 at 0x603000e8aa28 thread T9236
> #0 0x17ab9b3 in bool
> __gnu_cxx::__ops::_Iter_pred
> >::operator()<__gnu_cxx::__normal_iterator
> >(__gnu_cxx::__normal_iterator)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/predefined_ops.h:231:24
> #1 0x17ab745 in __gnu_cxx::__normal_iterator
> std::__find_if<__gnu_cxx::__normal_iterator,
> __gnu_cxx::__ops::_Iter_pred >
> >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator,
> __gnu_cxx::__ops::_Iter_pred >,
> std::random_access_iterator_tag)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/stl_algo.h:140:8
> #2 0x17ab2dc in __gnu_cxx::__normal_iterator
> std::__find_if<__gnu_cxx::__normal_iterator,
> __gnu_cxx::__ops::_Iter_pred >
> >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator,
> __gnu_cxx::__ops::_Iter_pred >)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/stl_algo.h:161:14
> #3 0x17aaf6c in __gnu_cxx::__normal_iterator
> std::find_if<__gnu_cxx::__normal_iterator,
> boost::algorithm::detail::is_any_ofF
> >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator,
> boost::algorithm::detail::is_any_ofF)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/stl_algo.h:3803:14
> #4 0x17aaba1 in boost::iterator_range<__gnu_cxx::__normal_iterator std::string> >
> boost::algorithm::detail::token_finderF
> >::operator()<__gnu_cxx::__normal_iterator
> >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator) const
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/detail/finder.hpp:565:41
> #5 0x17ac118 in
> boost::function2 std::string> >, __gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator
> >::operator()(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator) const
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #6 0x17abf8d in
> boost::algorithm::detail::find_iterator_base<__gnu_cxx::__normal_iterator std::string> >::do_find(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator) const
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/detail/find_iterator.hpp:63:32
> #7 0x17aa00c in
> boost::algorithm::split_iterator<__gnu_cxx::__normal_iterator std::string> >::increment()
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/find_iterator.hpp:305:44
> #8 0x17a95a5 in
> boost::algorithm::split_iterator<__gnu_cxx::__normal_iterator std::string>
> >::split_iterator
> > >(__gnu_cxx::__normal_iterator,
> __gnu_cxx::__normal_iterator,
> boost::algorithm::detail::token_finderF
> >)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/find_iterator.hpp:265:21
> #9 0x17a8d5e in std::vector >&
> boost::algorithm::iter_split std::allocator >, std::string,
> boost::algorithm::detail::token_finderF
> > >(std::vector >&, std::string&,
> boost::algorithm::detail::token_finderF
> >)
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/iter_find.hpp:170:21
> #10 0x179754f in std::vector >&
> boost::algorithm::split
> >, std::string, boost::algorithm::detail::is_any_ofF
> >(std::vector >&, std::string&,
> boost::algorithm::detail::is_any_ofF,
> boost::algorithm::token_compress_mode_type)
>
[jira] [Closed] (IMPALA-6642) start-impala-cluster.py failing in some custom cluster tests
[
https://issues.apache.org/jira/browse/IMPALA-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Taras Bobrovytsky closed IMPALA-6642.
-
Resolution: Cannot Reproduce
This issue was not seen for a while. Let's reopen it if it happens again.
> start-impala-cluster.py failing in some custom cluster tests
>
>
> Key: IMPALA-6642
> URL: https://issues.apache.org/jira/browse/IMPALA-6642
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.12.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Taras Bobrovytsky
>Priority: Critical
> Labels: broken-build
>
> Seen in recent builds, both on the 2.x branch. Not quite the same test/error,
> but they seem similar enough to probably be related:
> {noformat}
> 17:01:18 _ ERROR at setup of
> TestAdmissionController.test_require_user __
> 17:01:18 common/custom_cluster_test_suite.py:109: in setup_method
> 17:01:18 self._start_impala_cluster(cluster_args)
> 17:01:18 common/custom_cluster_test_suite.py:144: in _start_impala_cluster
> 17:01:18 check_call(cmd + options, close_fds=True)
> 17:01:18 /usr/lib64/python2.6/subprocess.py:505: in check_call
> 17:01:18 raise CalledProcessError(retcode, cmd)
> 17:01:18 E CalledProcessError: Command
> '['/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/bin/start-impala-cluster.py',
> '--cluster_size=3', '--num_coordinators=3',
> '--log_dir=/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests',
> '--log_level=1', '--impalad_args="-vmodule admission-controller=3
> -fair_scheduler_allocation_path
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/fair-scheduler-test2.xml
> -llama_site_path
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/llama-site-test2.xml
> -disable_admission_control=false -require_username" ',
> '--state_store_args="-statestore_heartbeat_frequency_ms=100
> -statestore_priority_update_frequency_ms=100" ']' returned non-zero exit
> status 1
> 17:01:18 Captured stdout setup
> -
> 17:01:18 Starting State Store logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 17:01:18 Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 17:01:18 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 17:01:18 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 17:01:18 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 17:01:18 Error starting cluster: Expected 3 impalad(s), only 2 found
> 17:01:18
> 17:01:18 Captured stderr setup
> -
> 17:01:18 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 17:01:18 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> {noformat}
> {noformat}
> 16:42:41 ___ ERROR at setup of
> TestAuthorization.test_access_runtime_profile
> 16:42:41 common/custom_cluster_test_suite.py:109: in setup_method
> 16:42:41 self._start_impala_cluster(cluster_args)
> 16:42:41 common/custom_cluster_test_suite.py:144: in _start_impala_cluster
> 16:42:41 check_call(cmd + options, close_fds=True)
> 16:42:41 /usr/lib64/python2.6/subprocess.py:505: in check_call
> 16:42:41 raise CalledProcessError(retcode, cmd)
> 16:42:41 E CalledProcessError: Command
> '['/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/bin/start-impala-cluster.py',
> '--cluster_size=3', '--num_coordinators=3',
> '--log_dir=/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests',
> '--log_level=1', '--impalad_args="--server_name=server1
> --authorization_policy_file=/test-warehouse/authz-policy.ini
> --authorized_proxy_user_config=hue=jenkins" ']' returned non-zero exit status
> 1
> 16:42:41 Captured stdout setup
> -
> 16:42:41 Starting State Store logging to
> /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 16:42:41 Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 16:42:41 Starting Impala Daemon logging to
>
[jira] [Created] (IMPALA-7177) Impalad hangs when starting up
Tianyi Wang created IMPALA-7177:
---
Summary: Impalad hangs when starting up
Key: IMPALA-7177
URL: https://issues.apache.org/jira/browse/IMPALA-7177
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Lars Volker
In
custom_cluster/test_breakpad.py::TestBreakpadExhaustive::test_minidump_cleanup_thread,
a impalad hangs for 4 minutes when starting up. The tail of its log is:
{noformat}
I0613 14:55:55.514082 23644 init.cc:246] OS version: Linux version
2.6.32-358.14.1.el6.centos.plus.x86_64 (mockbu...@c6b9.bsys.dev.centos.org)
(gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Tue Jul 16
21:33:24 UTC 2013
Clock: clocksource: 'xen', clockid_t: CLOCK_MONOTONIC_COARSE
I0613 14:55:55.514091 23644 init.cc:247] Process ID: 23644
I0613 14:55:55.514098 23644 init.cc:248] Default AES cipher mode for
spill-to-disk: AES-CTR
I0613 14:55:58.153287 23644 llvm-codegen.cc:152] CPU class for runtime code
generation: sandybridge
I0613 14:55:58.153379 23644 llvm-codegen.cc:154] Detected CPU flags:
+sse2,+cx16,-tbm,-avx512ifma,-avx512dq,-fma4,-prfchw,-bmi2,-xsavec,-fsgsbase,+popcnt,-aes,-xsaves,-avx512er,-avx512vpopcntdq,-clwb,-avx512f,-clzero,-pku,+mmx,-lwp$
I0613 14:55:58.153432 23644 llvm-codegen.cc:157] CPU flags enabled for runtime
code generation:
+sse2,+cx16,-tbm,-avx512ifma,-avx512dq,-fma4,-prfchw,-bmi2,-xsavec,-fsgsbase,+popcnt,-aes,-xsaves,-avx512er,-avx512vpopcntdq,-clwb,-avx$
I0613 14:56:00.633289 23644 GlogAppender.java:137] Logging (re)initialized.
Impala: VLOG, All other: INFO
I0613 14:56:00.638018 23644 JniFrontend.java:139] Authorization is 'DISABLED'.
I0613 14:56:00.640646 23644 JniFrontend.java:141] Java Input arguments:
-agentlib:jdwp=transport=dt_socket,address=30002,server=y,suspend=n
-Djava.library.path=/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/Impala-Toolchain/cdh_components/hadoop-3.0.0-cdh6.x-417234//lib/native/
-XX:ErrorFile=$
Java System properties:
awt.toolkit:sun.awt.X11.XToolkit
file.encoding.pkg:sun.io
java.specification.version:1.8
sun.cpu.isalist:
sun.jnu.encoding:UTF-8
java.class.path:/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/fe/src/test/resources:/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/fe/target/classes:/data/jenkins/workspace/impala-cdh6$
I0613 14:56:00.691942 23644 HiveConf.java:188] Found configuration file
file:/data/jenkins/workspace/impala-cdh6.x-exhaustive-release/repos/Impala/fe/src/test/resources/hive-site.xml
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6086) Use of permanent function should require SELECT privilege on DB
[
https://issues.apache.org/jira/browse/IMPALA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513154#comment-16513154
]
Zoram Thanga commented on IMPALA-6086:
--
Here's a rough diff for what I think should be done:
{code:java}
diff --git a/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
b/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
index 3e7f0cc..6256fdb 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
@@ -52,6 +52,7 @@ import com.google.common.base.Strings;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
+import com.google.common.collect.ImmutableList;
/**
* Wrapper class for parsing, analyzing and rewriting a SQL stmt.
*/
@@ -454,6 +455,9 @@ public class AnalysisContext {
List origColLabels =
Lists.newArrayList(analysisResult_.stmt_.getColLabels());
+ // Save old privileges
+ ImmutableList oldPrivReqs =
analysisResult_.analyzer_.getPrivilegeReqs();
+
// Re-analyze the stmt with a new analyzer.
analysisResult_.analyzer_ = createAnalyzer(stmtTableCache);
analysisResult_.stmt_.reset();
@@ -465,6 +469,11 @@ public class AnalysisContext {
if (LOG.isTraceEnabled()) {
LOG.trace("rewrittenStmt: " + analysisResult_.stmt_.toSql());
}
+
+ // Restore privilege requests found during the first pass
+ for (PrivilegeRequest req : oldPrivReqs) {
+analysisResult_.analyzer_.registerPrivReq(req);
+ }
if (isExplain) analysisResult_.stmt_.setIsExplain();
Preconditions.checkState(!analysisResult_.requiresSubqueryRewrite());
}
{code}
> Use of permanent function should require SELECT privilege on DB
> ---
>
> Key: IMPALA-6086
> URL: https://issues.apache.org/jira/browse/IMPALA-6086
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Security
>Affects Versions: Impala 2.9.0, Impala 3.1.0
>Reporter: Zoram Thanga
>Assignee: Zoram Thanga
>Priority: Minor
>
> A user that has no privilege on a database should not be able to execute any
> permanent functions in that database. This is currently possible, and should
> be fixed, so that the user must have SELECT privilege to execute permanent
> functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6086) Use of permanent function should require SELECT privilege on DB
[ https://issues.apache.org/jira/browse/IMPALA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513138#comment-16513138 ] Zoram Thanga commented on IMPALA-6086: -- Thanks for the inputs, [~tarmstrong], [~fredyw]. Agree trim() was probably a bad example. However, I have verified that the conclusion is still valid for a non-system UDF, for instance by creating the has_vowels() UDF example, as detailed [here|https://www.cloudera.com/documentation/enterprise/5-14-x/topics/impala_udf.html#udf_tutorial]. > Use of permanent function should require SELECT privilege on DB > --- > > Key: IMPALA-6086 > URL: https://issues.apache.org/jira/browse/IMPALA-6086 > Project: IMPALA > Issue Type: Bug > Components: Catalog, Security >Affects Versions: Impala 2.9.0, Impala 3.1.0 >Reporter: Zoram Thanga >Assignee: Zoram Thanga >Priority: Minor > > A user that has no privilege on a database should not be able to execute any > permanent functions in that database. This is currently possible, and should > be fixed, so that the user must have SELECT privilege to execute permanent > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-5604) Document DISABLE_CODEGEN_ROWS_THRESHOLD
[ https://issues.apache.org/jira/browse/IMPALA-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong reassigned IMPALA-5604: - Assignee: Tim Armstrong (was: Alex Rodoni) > Document DISABLE_CODEGEN_ROWS_THRESHOLD > --- > > Key: IMPALA-5604 > URL: https://issues.apache.org/jira/browse/IMPALA-5604 > Project: IMPALA > Issue Type: Task > Components: Docs >Affects Versions: Impala 2.10.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > > We just checking in DISABLE_CODEGEN_ROWS_THRESHOLD to master with default > value 50,000. > The option uses the planner's estimates to determine whether to disable > codegen for a query based on the maximum number of rows flowing through any > part of the query plan. > It is similar to EXEC_SINGLE_NODE_ROWS_THRESHOLD in behaviour except the > threshold is per-node rows instead of total rows. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6086) Use of permanent function should require SELECT privilege on DB
[ https://issues.apache.org/jira/browse/IMPALA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513134#comment-16513134 ] Fredy Wijaya commented on IMPALA-6086: -- In this particular example, trim() is a built-in function that lives in a system database (_impala_builtins), which requires no special privilege. See [https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java#L569-L573] We need to find another example with a UDF in a non-system database that gets constant folded and to see if the privilege requests get reset. > Use of permanent function should require SELECT privilege on DB > --- > > Key: IMPALA-6086 > URL: https://issues.apache.org/jira/browse/IMPALA-6086 > Project: IMPALA > Issue Type: Bug > Components: Catalog, Security >Affects Versions: Impala 2.9.0, Impala 3.1.0 >Reporter: Zoram Thanga >Assignee: Zoram Thanga >Priority: Minor > > A user that has no privilege on a database should not be able to execute any > permanent functions in that database. This is currently possible, and should > be fixed, so that the user must have SELECT privilege to execute permanent > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6086) Use of permanent function should require SELECT privilege on DB
[ https://issues.apache.org/jira/browse/IMPALA-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513098#comment-16513098 ] Tim Armstrong commented on IMPALA-6086: --- [~zoram] I left you hanging for a while here... yeah it seems like this is maybe related to the lifecycle of query rewrites and authorisation requests. It seems like we shouldn't execute the function at all, even for the purposes of constant folding, if the user doesn't have permissions. IIRC we do the full analysis, collect authorisation checks, and then check privileges only after analysis completes: https://github.com/apache/impala/blob/1d699a142b2b4adc0dc19665f6196ffc1eff4bfa/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java#L416 [~aholley][~fredyw] this may be of interest. > Use of permanent function should require SELECT privilege on DB > --- > > Key: IMPALA-6086 > URL: https://issues.apache.org/jira/browse/IMPALA-6086 > Project: IMPALA > Issue Type: Bug > Components: Catalog, Security >Affects Versions: Impala 2.9.0, Impala 3.1.0 >Reporter: Zoram Thanga >Assignee: Zoram Thanga >Priority: Minor > > A user that has no privilege on a database should not be able to execute any > permanent functions in that database. This is currently possible, and should > be fixed, so that the user must have SELECT privilege to execute permanent > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-5552) Proxy user list should support groups
[
https://issues.apache.org/jira/browse/IMPALA-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fredy Wijaya resolved IMPALA-5552.
--
Resolution: Fixed
Fix Version/s: Impala 3.1.0
Impala 2.13.0
> Proxy user list should support groups
> -
>
> Key: IMPALA-5552
> URL: https://issues.apache.org/jira/browse/IMPALA-5552
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Reporter: Tristan Stevens
>Assignee: Fredy Wijaya
>Priority: Critical
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> The authorized_proxy_user_config takes a map of user->doAsUser* - i.e. user
> is allowed to impersonate any users in the list of doAsUsers.
> For enterprise deployments, this would be better specified as a list of
> groups, rather than a a list of users:
> user1->group*
> When accepting a query, Impala will check that the doAs user is a member of
> any of the list of groups specified for the connecting user.
> HiveServer2 does this via Hadoop-level proxy user privileges (e.g.
> {{
> hadoop.proxyuser.user1.hosts
> doAsUser1,doAsUser2
>
>
> hadoop.proxyuser.user1.groups
> doAsGroup1,doAsGroup2
> }}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7174) TestAdmissionController.test_cancellation failed with incorrect total-admitted metric
[
https://issues.apache.org/jira/browse/IMPALA-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7174:
--
Priority: Blocker (was: Critical)
> TestAdmissionController.test_cancellation failed with incorrect
> total-admitted metric
> -
>
> Key: IMPALA-7174
> URL: https://issues.apache.org/jira/browse/IMPALA-7174
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: broken-build
> Attachments:
> impalad.ec2-m2-4xlarge-centos-6-4-01f7.vpc.cloudera.com.jenkins.log.INFO.20180614-060607.2553
>
>
> The failed revision is ee9a9b6c5000cf915716a15ea8a0b3605290a9a5, an
> descendant of 'IMPALA-5216: Make admission control queuing async'.
> {noformat}
> Stacktrace
> custom_cluster/test_admission_controller.py:557: in test_cancellation
> assert self.cluster.impalads[0].service.get_metric_value(
> E assert 0 == 3
> E+ where 0 = 0x4ed1b90>>('admission-controller.total-admitted.default-pool')
> E+where > =
> 0x4ed1b90>.get_metric_value
> E+ where 0x4ed1b90> = 0x52474d0>.service
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7174) TestAdmissionController.test_cancellation failed with incorrect total-admitted metric
[
https://issues.apache.org/jira/browse/IMPALA-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513044#comment-16513044
]
Tim Armstrong commented on IMPALA-7174:
---
Hrm, it's rejected for a different reason.
{noformat}
I0614 06:06:19.532265 3227 Frontend.java:997] Analyzing query: select 1
I0614 06:06:19.544095 2861 admission-controller.cc:670] Full
impala-request-queue stats update
I0614 06:06:19.636790 3227 Frontend.java:1009] Analysis finished.
I0614 06:06:19.644654 2859 admission-controller.cc:670] Full
impala-request-queue stats update
I0614 06:06:19.744290 2861 admission-controller.cc:670] Full
impala-request-queue stats update
I0614 06:06:19.832564 3230 admission-controller.cc:521] Schedule for
id=a74d28adf74b3759:94628ae1 in pool_name=default-pool
cluster_mem_needed=1.00 GB PoolConfig: max_requests=1 max_queued=1 max_mem=1.00
GB
I0614 06:06:19.832835 3230 admission-controller.cc:526] Stats:
agg_num_running=0, agg_num_queued=0, agg_mem_reserved=0,
local_host(local_mem_admitted=0, num_admitted_running=0, num_queued=0,
backend_mem_reserved=0)
I0614 06:06:19.833416 3230 admission-controller.cc:542] Rejected query from
pool default-pool: request memory needed 1.00 GB is greater than pool max mem
resources 1.00 GB.
Use the MEM_LIMIT query option to indicate how much memory is required per
node. The total memory needed is the per-node MEM_LIMIT times the number of
nodes executing the query. See the Admission Control documentation for more
information.
{noformat}
Oh, duh, the sleeps are wrapped in #ifndef NDEBUG and this is a release build...
> TestAdmissionController.test_cancellation failed with incorrect
> total-admitted metric
> -
>
> Key: IMPALA-7174
> URL: https://issues.apache.org/jira/browse/IMPALA-7174
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build
> Attachments:
> impalad.ec2-m2-4xlarge-centos-6-4-01f7.vpc.cloudera.com.jenkins.log.INFO.20180614-060607.2553
>
>
> The failed revision is ee9a9b6c5000cf915716a15ea8a0b3605290a9a5, an
> descendant of 'IMPALA-5216: Make admission control queuing async'.
> {noformat}
> Stacktrace
> custom_cluster/test_admission_controller.py:557: in test_cancellation
> assert self.cluster.impalads[0].service.get_metric_value(
> E assert 0 == 3
> E+ where 0 = 0x4ed1b90>>('admission-controller.total-admitted.default-pool')
> E+where > =
> 0x4ed1b90>.get_metric_value
> E+ where 0x4ed1b90> = 0x52474d0>.service
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7174) TestAdmissionController.test_cancellation failed with incorrect total-admitted metric
[
https://issues.apache.org/jira/browse/IMPALA-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7174:
--
Attachment:
impalad.ec2-m2-4xlarge-centos-6-4-01f7.vpc.cloudera.com.jenkins.log.INFO.20180614-060607.2553
> TestAdmissionController.test_cancellation failed with incorrect
> total-admitted metric
> -
>
> Key: IMPALA-7174
> URL: https://issues.apache.org/jira/browse/IMPALA-7174
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build
> Attachments:
> impalad.ec2-m2-4xlarge-centos-6-4-01f7.vpc.cloudera.com.jenkins.log.INFO.20180614-060607.2553
>
>
> The failed revision is ee9a9b6c5000cf915716a15ea8a0b3605290a9a5, an
> descendant of 'IMPALA-5216: Make admission control queuing async'.
> {noformat}
> Stacktrace
> custom_cluster/test_admission_controller.py:557: in test_cancellation
> assert self.cluster.impalads[0].service.get_metric_value(
> E assert 0 == 3
> E+ where 0 = 0x4ed1b90>>('admission-controller.total-admitted.default-pool')
> E+where > =
> 0x4ed1b90>.get_metric_value
> E+ where 0x4ed1b90> = 0x52474d0>.service
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7174) TestAdmissionController.test_cancellation failed with incorrect total-admitted metric
[
https://issues.apache.org/jira/browse/IMPALA-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513009#comment-16513009
]
Tim Armstrong commented on IMPALA-7174:
---
It looks like this failed because of a caught exception (I missed the wonky
exception handling when reviewing)
{noformat}
("Expected 1 lines in file
/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests/impalad.ec2-m2-4xlarge-centos-6-4-01f7.vpc.cloudera.com.jenkins.log.INFO.20180614-060607.2553
matching regex 'Ready to be Rejected but already cancelled, query id=', but
found 0 lines. Last line was: \nI0614 06:06:20.849289 2859
admission-controller.cc:670] Full impala-request-queue stats update\n",)
{noformat}
I don't have reason to think that it's a product bug, so I'll xfail when we
catch an exception until we can work out the proper fix.
> TestAdmissionController.test_cancellation failed with incorrect
> total-admitted metric
> -
>
> Key: IMPALA-7174
> URL: https://issues.apache.org/jira/browse/IMPALA-7174
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build
>
> The failed revision is ee9a9b6c5000cf915716a15ea8a0b3605290a9a5, an
> descendant of 'IMPALA-5216: Make admission control queuing async'.
> {noformat}
> Stacktrace
> custom_cluster/test_admission_controller.py:557: in test_cancellation
> assert self.cluster.impalads[0].service.get_metric_value(
> E assert 0 == 3
> E+ where 0 = 0x4ed1b90>>('admission-controller.total-admitted.default-pool')
> E+where > =
> 0x4ed1b90>.get_metric_value
> E+ where 0x4ed1b90> = 0x52474d0>.service
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7076) Impala 2.13 & 3.1 Docs: Object ownership for Impala
[ https://issues.apache.org/jira/browse/IMPALA-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7076: Target Version: Impala 2.13.0, Impala 3.1.0 > Impala 2.13 & 3.1 Docs: Object ownership for Impala > --- > > Key: IMPALA-7076 > URL: https://issues.apache.org/jira/browse/IMPALA-7076 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Fredy Wijaya >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7076) Impala 2.13 & 3.1 Docs: Object ownership for Impala
[ https://issues.apache.org/jira/browse/IMPALA-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7076: Summary: Impala 2.13 & 3.1 Docs: Object ownership for Impala (was: [DOCS] Object ownership for Impala) > Impala 2.13 & 3.1 Docs: Object ownership for Impala > --- > > Key: IMPALA-7076 > URL: https://issues.apache.org/jira/browse/IMPALA-7076 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Fredy Wijaya >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6352) TestTableSample took too long in recent tests
[
https://issues.apache.org/jira/browse/IMPALA-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-6352:
-
Assignee: (was: Tim Armstrong)
> TestTableSample took too long in recent tests
> -
>
> Key: IMPALA-6352
> URL: https://issues.apache.org/jira/browse/IMPALA-6352
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.12.0
>Reporter: Vuk Ercegovac
>Priority: Critical
> Labels: broken-build
>
> TestTableSample test took ~8 hours in recent (12/21) exhaustive rhel tests.
> That caused the overall test to be aborted:
> ...
> 01:53:10 [gw2] PASSED
> query_test/test_tablesample.py::TestTableSample::test_tablesample[repeatable:
> True | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> seq/gzip/block]
> 01:53:10
> query_test/test_tablesample.py::TestTableSample::test_tablesample[repeatable:
> False | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> seq/gzip/block]
> 10:03:51 [gw2] PASSED
> query_test/test_tablesample.py::TestTableSample::test_tablesample[repeatable:
> False | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> seq/gzip/block] Build timed out (after 1,440 minutes). Marking the build as
> aborted.
> 10:03:51 Build was aborted
> ...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Reopened] (IMPALA-6917) Implement COMMENT ON TABLE/VIEW
[
https://issues.apache.org/jira/browse/IMPALA-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fredy Wijaya reopened IMPALA-6917:
--
Accidentally pressed resolved. Re-opning it for now. GVO is currently running.
> Implement COMMENT ON TABLE/VIEW
> ---
>
> Key: IMPALA-6917
> URL: https://issues.apache.org/jira/browse/IMPALA-6917
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
>Reporter: Fredy Wijaya
>Assignee: Fredy Wijaya
>Priority: Minor
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> Syntax:
> {noformat}
> COMMENT ON TABLE my_db.my_table IS 'Employee Information';
> COMMENT ON VIEW my_db.my_view IS 'Employee Information';{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7016) Statement to allow setting ownership for database
[
https://issues.apache.org/jira/browse/IMPALA-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fredy Wijaya resolved IMPALA-7016.
--
Resolution: Fixed
Fix Version/s: Impala 3.1.0
Impala 2.13.0
> Statement to allow setting ownership for database
> -
>
> Key: IMPALA-7016
> URL: https://issues.apache.org/jira/browse/IMPALA-7016
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
>Affects Versions: Impala 3.0, Impala 2.13.0
>Reporter: Adam Holley
>Assignee: Fredy Wijaya
>Priority: Major
> Labels: security
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> Create statement to allow setting owner on database
> {noformat}
> ALTER DATABASE database_name SET OWNER [USER|ROLE] user_or_role;{noformat}
> Examples:
> {noformat}
> ALTER DATABASE SET OWNER USER
> ALTER DATABASE SET OWNER ROLE
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-6917) Implement COMMENT ON TABLE/VIEW
[
https://issues.apache.org/jira/browse/IMPALA-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fredy Wijaya resolved IMPALA-6917.
--
Resolution: Fixed
Fix Version/s: Impala 3.1.0
Impala 2.13.0
> Implement COMMENT ON TABLE/VIEW
> ---
>
> Key: IMPALA-6917
> URL: https://issues.apache.org/jira/browse/IMPALA-6917
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
>Reporter: Fredy Wijaya
>Assignee: Fredy Wijaya
>Priority: Minor
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> Syntax:
> {noformat}
> COMMENT ON TABLE my_db.my_table IS 'Employee Information';
> COMMENT ON VIEW my_db.my_view IS 'Employee Information';{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-6788) Query fragments can spend lots of time starting up then fail right after "starting" all backends
[
https://issues.apache.org/jira/browse/IMPALA-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-6788 started by Dan Hecht.
-
> Query fragments can spend lots of time starting up then fail right after
> "starting" all backends
>
>
> Key: IMPALA-6788
> URL: https://issues.apache.org/jira/browse/IMPALA-6788
> Project: IMPALA
> Issue Type: Sub-task
> Components: Distributed Exec
>Affects Versions: Impala 2.12.0
>Reporter: Mostafa Mokhtar
>Assignee: Dan Hecht
>Priority: Major
> Labels: krpc, rpc
> Attachments: connect_thread_busy_queries_failing.txt,
> impalad.va1007.foo.com.impala.log.INFO.20180401-200453.1800807.zip
>
>
> Logs from a large cluster show that query startup can take a long time, then
> once the startup completes the query is cancelled, this is because one of the
> intermediate rpcs failed.
> Not clear what the right answer is as fragments are started asynchronously,
> possibly a timeout?
> {code}
> I0401 21:25:30.776803 1830900 coordinator.cc:99] Exec()
> query_id=334cc7dd9758c36c:ec38aeb4 stmt=with customer_total_return as
> I0401 21:25:30.813993 1830900 coordinator.cc:357] starting execution on 644
> backends for query_id=334cc7dd9758c36c:ec38aeb4
> I0401 21:29:58.406466 1830900 coordinator.cc:370] started execution on 644
> backends for query_id=334cc7dd9758c36c:ec38aeb4
> I0401 21:29:58.412132 1830900 coordinator.cc:896] Cancel()
> query_id=334cc7dd9758c36c:ec38aeb4
> I0401 21:29:59.188817 1830900 coordinator.cc:906] CancelBackends()
> query_id=334cc7dd9758c36c:ec38aeb4, tried to cancel 643 backends
> I0401 21:29:59.189177 1830900 coordinator.cc:1092] Release admission control
> resources for query_id=334cc7dd9758c36c:ec38aeb4
> {code}
> {code}
> I0401 21:23:48.218379 1830386 coordinator.cc:99] Exec()
> query_id=e44d553b04d47cfb:28f06bb8 stmt=with customer_total_return as
> I0401 21:23:48.270226 1830386 coordinator.cc:357] starting execution on 640
> backends for query_id=e44d553b04d47cfb:28f06bb8
> I0401 21:29:58.402195 1830386 coordinator.cc:370] started execution on 640
> backends for query_id=e44d553b04d47cfb:28f06bb8
> I0401 21:29:58.403818 1830386 coordinator.cc:896] Cancel()
> query_id=e44d553b04d47cfb:28f06bb8
> I0401 21:29:59.255903 1830386 coordinator.cc:906] CancelBackends()
> query_id=e44d553b04d47cfb:28f06bb8, tried to cancel 639 backends
> I0401 21:29:59.256251 1830386 coordinator.cc:1092] Release admission control
> resources for query_id=e44d553b04d47cfb:28f06bb8
> {code}
> Checked the coordinator and threads appear to be spending lots of time
> waiting on exec_complete_barrier_
> {code}
> #0 0x7fd928c816d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1 0x01222944 in impala::Promise::Get() ()
> #2 0x01220d7b in impala::Coordinator::StartBackendExec() ()
> #3 0x01221c87 in impala::Coordinator::Exec() ()
> #4 0x00c3a925 in
> impala::ClientRequestState::ExecQueryOrDmlRequest(impala::TQueryExecRequest
> const&) ()
> #5 0x00c41f7e in
> impala::ClientRequestState::Exec(impala::TExecRequest*) ()
> #6 0x00bff597 in
> impala::ImpalaServer::ExecuteInternal(impala::TQueryCtx const&,
> std::shared_ptr, bool*,
> std::shared_ptr*) ()
> #7 0x00c061d9 in impala::ImpalaServer::Execute(impala::TQueryCtx*,
> std::shared_ptr,
> std::shared_ptr*) ()
> #8 0x00c561c5 in impala::ImpalaServer::query(beeswax::QueryHandle&,
> beeswax::Query const&) ()
> /StartBackendExec
> #11 0x00d60c9a in boost::detail::thread_data void (*)(std::string const&, std::string const&, boost::function,
> impala::ThreadDebugInfo const*, impala::Promise*),
> boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> > > >::run() ()
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7046) Add targeted regression test for race in IMPALA-7033
[ https://issues.apache.org/jira/browse/IMPALA-7046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-7046 started by Dan Hecht. - > Add targeted regression test for race in IMPALA-7033 > > > Key: IMPALA-7046 > URL: https://issues.apache.org/jira/browse/IMPALA-7046 > Project: IMPALA > Issue Type: Task > Components: Backend >Affects Versions: Impala 3.1.0 >Reporter: Dan Hecht >Assignee: Dan Hecht >Priority: Major > > I'd like to add a regression test to trigger the race in IMPALA-7033 more > reliably, but it will involve doing some sleeps at specific places, so I'd > like to add it after [~bikramjeet.vig] commits a change that provides some > infrastructure for that. > The race was: > 1) Coordinator::Exec() takes the QueryState ExecResources reference count. > 2) Coordinator sends out exec rpc to non-coordinator backend. > 3) Some non-coordinator backend sends a failure report which invokes > HandleExecStateTransition, which drops the coordinator's reference to the > exec resources. > 4) Coordinator sends out exec rpc to coordinator backend, which takes the > exec resources reference and releases it. We don't expect the reference count > to become non-zero after it has already gone through a cycle. > The fix for this race is included in [https://gerrit.cloudera.org/#/c/10440] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-6942) "Cancelled due to unreachable impalad(s)" error message is misleading
[ https://issues.apache.org/jira/browse/IMPALA-6942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-6942 started by Dan Hecht. - > "Cancelled due to unreachable impalad(s)" error message is misleading > - > > Key: IMPALA-6942 > URL: https://issues.apache.org/jira/browse/IMPALA-6942 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.0 >Reporter: Dan Hecht >Assignee: Dan Hecht >Priority: Major > > The error message "Cancelled due to unreachable impalad(s)" would be better > as "Failed due to unreachable impalad(s)" since the query has failed. The > code happens to trigger cancellation via the same path as client > cancellation, but this is really a query failure. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7173) [DOCS] Update load balancer configuration in our docs
[ https://issues.apache.org/jira/browse/IMPALA-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Brown resolved IMPALA-7173. --- Resolution: Duplicate Fix Version/s: Impala 2.12.0 Dupe of IMPALA-6515. It's already fixed, even. :) http://impala.apache.org/docs/build/html/topics/impala_proxy.html > [DOCS] Update load balancer configuration in our docs > - > > Key: IMPALA-7173 > URL: https://issues.apache.org/jira/browse/IMPALA-7173 > Project: IMPALA > Issue Type: Documentation > Components: Docs >Affects Versions: Impala 2.11.0 >Reporter: Laurel Hale >Assignee: Laurel Hale >Priority: Minor > Fix For: Impala 2.12.0 > > > This Jira is filed on behalf of Sudarshan Sharma: > 1) Please refer following documentation at > [http://impala.apache.org/docs/build/html/topics/impala_proxy.html#tut_proxy] > . > 2) Example configuration has following. > listen impala :25003 > mode tcp > option tcplog > balance leastconn > server symbolic_name_1 impala-host-1.example.com:21000 > server symbolic_name_2 impala-host-2.example.com:21000 > server symbolic_name_3 impala-host-3.example.com:21000 > server symbolic_name_4 impala-host-4.example.com:21000 > ** > Request :- > ** > There should be a "check" keyword at the end of each line that begins with > "server". Can we please add? (most users are likely to copy - paste and > replication and hence it would be best to keep it correct) > This is correct for the following pool mentioned later in the configuration. > > listen impalajdbc :21051 > mode tcp > option tcplog > balance source > server symbolic_name_5 impala-host-1.example.com:21050 check > server symbolic_name_6 impala-host-2.example.com:21050 check > server symbolic_name_7 impala-host-3.example.com:21050 check > server symbolic_name_8 impala-host-4.example.com:21050 check -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7173) [DOCS] Update load balancer configuration in our docs
[ https://issues.apache.org/jira/browse/IMPALA-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni reassigned IMPALA-7173: --- Assignee: Laurel Hale (was: Alex Rodoni) > [DOCS] Update load balancer configuration in our docs > - > > Key: IMPALA-7173 > URL: https://issues.apache.org/jira/browse/IMPALA-7173 > Project: IMPALA > Issue Type: Documentation > Components: Docs >Affects Versions: Impala 2.11.0 >Reporter: Laurel Hale >Assignee: Laurel Hale >Priority: Minor > > This Jira is filed on behalf of Sudarshan Sharma: > 1) Please refer following documentation at > [http://impala.apache.org/docs/build/html/topics/impala_proxy.html#tut_proxy] > . > 2) Example configuration has following. > listen impala :25003 > mode tcp > option tcplog > balance leastconn > server symbolic_name_1 impala-host-1.example.com:21000 > server symbolic_name_2 impala-host-2.example.com:21000 > server symbolic_name_3 impala-host-3.example.com:21000 > server symbolic_name_4 impala-host-4.example.com:21000 > ** > Request :- > ** > There should be a "check" keyword at the end of each line that begins with > "server". Can we please add? (most users are likely to copy - paste and > replication and hence it would be best to keep it correct) > This is correct for the following pool mentioned later in the configuration. > > listen impalajdbc :21051 > mode tcp > option tcplog > balance source > server symbolic_name_5 impala-host-1.example.com:21050 check > server symbolic_name_6 impala-host-2.example.com:21050 check > server symbolic_name_7 impala-host-3.example.com:21050 check > server symbolic_name_8 impala-host-4.example.com:21050 check -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7162) [DOCS] idle_query_timeout setting in Impala needs more clarification
[ https://issues.apache.org/jira/browse/IMPALA-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni reassigned IMPALA-7162: --- Assignee: Laurel Hale > [DOCS] idle_query_timeout setting in Impala needs more clarification > > > Key: IMPALA-7162 > URL: https://issues.apache.org/jira/browse/IMPALA-7162 > Project: IMPALA > Issue Type: Documentation > Components: Docs >Affects Versions: Impala 2.6.0 >Reporter: Laurel Hale >Assignee: Laurel Hale >Priority: Minor > Labels: documentation > > This Jira is created on behalf of Sudarshan Sharma: > 1) Please refer to the following Impala documentation: > [http://impala.apache.org/docs/build3x/html/topics/impala_timeouts.html] > 2) Please see the section where idle_query_timeout is explained. > * > Request:- > * > We've mentioned that idle_query_timeout cancels the query. > Can we please elaborate it further and mention that query still stays open > (after being cancelled) and consumes a slot in Impala admission control? > Therefore, though idle_query_timeout helps in saving some resources at Impala > end, you may still want to configure idle_Session_timeout to same value since > idle_Session_timeout even closes the query for idle sessions (instead of just > cancelling) though the requirement for it to trigger is session should stay > idle. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7175) In a local FS build, test_native_functions_race thinks there are 2 impalads where there should be 1
[
https://issues.apache.org/jira/browse/IMPALA-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang updated IMPALA-7175:
Summary: In a local FS build, test_native_functions_race thinks there are 2
impalads where there should be 1 (was: In a local build,
test_native_functions_race thinks there are 2 impalads where there should be 1)
> In a local FS build, test_native_functions_race thinks there are 2 impalads
> where there should be 1
> ---
>
> Key: IMPALA-7175
> URL: https://issues.apache.org/jira/browse/IMPALA-7175
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Vuk Ercegovac
>Priority: Critical
> Labels: broken-build
>
> In TestUdfExecution.test_native_functions_race, the test checks the number of
> impalads at the beginning and end of the test. In a local build there should
> be only 1 impalad but somehow the test found 2 at the beginning of the test
> and failed.
> {noformat}
> Stacktrace
> query_test/test_udfs.py:379: in test_native_functions_race
> assert len(cluster.impalads) == exp_num_impalads
> E assert 1 == 2
> E+ where 1 = len([ 0xc9ffa90>])
> E+where [ 0xc9ffa90>] = 0x6a5d510>.impalads
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7175) In a local build, test_native_functions_race thinks there are 2 impalads where there should be 1
Tianyi Wang created IMPALA-7175:
---
Summary: In a local build, test_native_functions_race thinks there
are 2 impalads where there should be 1
Key: IMPALA-7175
URL: https://issues.apache.org/jira/browse/IMPALA-7175
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Vuk Ercegovac
In TestUdfExecution.test_native_functions_race, the test checks the number of
impalads at the beginning and end of the test. In a local build there should be
only 1 impalad but somehow the test found 2 at the beginning of the test and
failed.
{noformat}
Stacktrace
query_test/test_udfs.py:379: in test_native_functions_race
assert len(cluster.impalads) == exp_num_impalads
E assert 1 == 2
E+ where 1 = len([])
E+where [] = .impalads
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7173) [DOCS] Update load balancer configuration in our docs
[ https://issues.apache.org/jira/browse/IMPALA-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512885#comment-16512885 ] Alex Rodoni commented on IMPALA-7173: - [~achoi] Do you approve the change suggested? > [DOCS] Update load balancer configuration in our docs > - > > Key: IMPALA-7173 > URL: https://issues.apache.org/jira/browse/IMPALA-7173 > Project: IMPALA > Issue Type: Documentation > Components: Docs >Affects Versions: Impala 2.11.0 >Reporter: Laurel Hale >Assignee: Alex Rodoni >Priority: Minor > > This Jira is filed on behalf of Sudarshan Sharma: > 1) Please refer following documentation at > [http://impala.apache.org/docs/build/html/topics/impala_proxy.html#tut_proxy] > . > 2) Example configuration has following. > listen impala :25003 > mode tcp > option tcplog > balance leastconn > server symbolic_name_1 impala-host-1.example.com:21000 > server symbolic_name_2 impala-host-2.example.com:21000 > server symbolic_name_3 impala-host-3.example.com:21000 > server symbolic_name_4 impala-host-4.example.com:21000 > ** > Request :- > ** > There should be a "check" keyword at the end of each line that begins with > "server". Can we please add? (most users are likely to copy - paste and > replication and hence it would be best to keep it correct) > This is correct for the following pool mentioned later in the configuration. > > listen impalajdbc :21051 > mode tcp > option tcplog > balance source > server symbolic_name_5 impala-host-1.example.com:21050 check > server symbolic_name_6 impala-host-2.example.com:21050 check > server symbolic_name_7 impala-host-3.example.com:21050 check > server symbolic_name_8 impala-host-4.example.com:21050 check -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7173) [DOCS] Update load balancer configuration in our docs
[ https://issues.apache.org/jira/browse/IMPALA-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni reassigned IMPALA-7173: --- Assignee: Alex Rodoni > [DOCS] Update load balancer configuration in our docs > - > > Key: IMPALA-7173 > URL: https://issues.apache.org/jira/browse/IMPALA-7173 > Project: IMPALA > Issue Type: Documentation > Components: Docs >Affects Versions: Impala 2.11.0 >Reporter: Laurel Hale >Assignee: Alex Rodoni >Priority: Minor > > This Jira is filed on behalf of Sudarshan Sharma: > 1) Please refer following documentation at > [http://impala.apache.org/docs/build/html/topics/impala_proxy.html#tut_proxy] > . > 2) Example configuration has following. > listen impala :25003 > mode tcp > option tcplog > balance leastconn > server symbolic_name_1 impala-host-1.example.com:21000 > server symbolic_name_2 impala-host-2.example.com:21000 > server symbolic_name_3 impala-host-3.example.com:21000 > server symbolic_name_4 impala-host-4.example.com:21000 > ** > Request :- > ** > There should be a "check" keyword at the end of each line that begins with > "server". Can we please add? (most users are likely to copy - paste and > replication and hence it would be best to keep it correct) > This is correct for the following pool mentioned later in the configuration. > > listen impalajdbc :21051 > mode tcp > option tcplog > balance source > server symbolic_name_5 impala-host-1.example.com:21050 check > server symbolic_name_6 impala-host-2.example.com:21050 check > server symbolic_name_7 impala-host-3.example.com:21050 check > server symbolic_name_8 impala-host-4.example.com:21050 check -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7174) TestAdmissionController.test_cancellation failed with incorrect total-admitted metric
Tianyi Wang created IMPALA-7174:
---
Summary: TestAdmissionController.test_cancellation failed with
incorrect total-admitted metric
Key: IMPALA-7174
URL: https://issues.apache.org/jira/browse/IMPALA-7174
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.1.0
Reporter: Tianyi Wang
Assignee: Tim Armstrong
The failed revision is ee9a9b6c5000cf915716a15ea8a0b3605290a9a5, an descendant
of 'IMPALA-5216: Make admission control queuing async'.
{noformat}
Stacktrace
custom_cluster/test_admission_controller.py:557: in test_cancellation
assert self.cluster.impalads[0].service.get_metric_value(
E assert 0 == 3
E+ where 0 = >('admission-controller.total-admitted.default-pool')
E+where > =
.get_metric_value
E+ where = .service
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7171) Add docs for Kudu insert partitioning/sorting
[
https://issues.apache.org/jira/browse/IMPALA-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512853#comment-16512853
]
Alex Rodoni commented on IMPALA-7171:
-
[~twmarshall] In impala_hints.html, we have:
Starting from Impala 2.9, {{INSERT}} or {{UPSERT}} operations into Kudu tables
automatically have an exchange and sort node added to the plan that partitions
and sorts the rows according to the partitioning/primary key scheme of the
target table (unless the number of rows to be inserted is small enough to
trigger single node execution). Use the{{ /* +NOCLUSTERED */}} and {{/*
+NOSHUFFLE */}} hints together to disable partitioning and sorting before the
rows are sent to Kudu.
Do you want to use the same text in impala_kudu.html?
> Add docs for Kudu insert partitioning/sorting
> -
>
> Key: IMPALA-7171
> URL: https://issues.apache.org/jira/browse/IMPALA-7171
> Project: IMPALA
> Issue Type: Improvement
> Components: Docs
>Reporter: Thomas Tauber-Marshall
>Assignee: Alex Rodoni
>Priority: Major
> Labels: docs
>
> On the page:
> http://impala.apache.org/docs/build3x/html/topics/impala_kudu.html, at the
> end of the section: "Impala DML Support for Kudu Tables (INSERT, UPDATE,
> DELETE, UPSERT)", we should add text like:
> Starting from Impala 2.9, Impala will automatically add a partition and sort
> step to INSERTs before sending the rows to Kudu. Since Kudu partitions and
> sorts rows on write, pre-partitioning and sorting takes some of the load off
> of Kudu, and helps ensure that large INSERTs complete without timing out, but
> it may slow down the end-to-end performance of the INSERT. Starting from
> Impala 2.10, the hints "/* +noshuffle,noclustered */" may be used to turn
> this pre-partitioning and sorting off. Additionally, since sorting may
> consume a lot of memory, users should consider setting a "mem_limit" for
> these queries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7173) [DOCS] Update load balancer configuration in our docs
[ https://issues.apache.org/jira/browse/IMPALA-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Apple updated IMPALA-7173: -- Description: This Jira is filed on behalf of Sudarshan Sharma: 1) Please refer following documentation at [http://impala.apache.org/docs/build/html/topics/impala_proxy.html#tut_proxy] . 2) Example configuration has following. listen impala :25003 mode tcp option tcplog balance leastconn server symbolic_name_1 impala-host-1.example.com:21000 server symbolic_name_2 impala-host-2.example.com:21000 server symbolic_name_3 impala-host-3.example.com:21000 server symbolic_name_4 impala-host-4.example.com:21000 ** Request :- ** There should be a "check" keyword at the end of each line that begins with "server". Can we please add? (most users are likely to copy - paste and replication and hence it would be best to keep it correct) This is correct for the following pool mentioned later in the configuration. listen impalajdbc :21051 mode tcp option tcplog balance source server symbolic_name_5 impala-host-1.example.com:21050 check server symbolic_name_6 impala-host-2.example.com:21050 check server symbolic_name_7 impala-host-3.example.com:21050 check server symbolic_name_8 impala-host-4.example.com:21050 check was: This Jira is filed on behalf of Sudarshan Sharma: 1) Please refer following documentation at [https://www.cloudera.com/documentation/enterprise/5-14-x/topics/impala_proxy.html#tut_proxy] . 2) Example configuration has following. listen impala :25003 mode tcp option tcplog balance leastconn server symbolic_name_1 impala-host-1.example.com:21000 server symbolic_name_2 impala-host-2.example.com:21000 server symbolic_name_3 impala-host-3.example.com:21000 server symbolic_name_4 impala-host-4.example.com:21000 ** Request :- ** There should be a "check" keyword at the end of each line that begins with "server". Can we please add? (most users are likely to copy - paste and replication and hence it would be best to keep it correct) This is correct for the following pool mentioned later in the configuration. listen impalajdbc :21051 mode tcp option tcplog balance source server symbolic_name_5 impala-host-1.example.com:21050 check server symbolic_name_6 impala-host-2.example.com:21050 check server symbolic_name_7 impala-host-3.example.com:21050 check server symbolic_name_8 impala-host-4.example.com:21050 check > [DOCS] Update load balancer configuration in our docs > - > > Key: IMPALA-7173 > URL: https://issues.apache.org/jira/browse/IMPALA-7173 > Project: IMPALA > Issue Type: Documentation > Components: Docs >Affects Versions: Impala 2.11.0 >Reporter: Laurel Hale >Priority: Minor > > This Jira is filed on behalf of Sudarshan Sharma: > 1) Please refer following documentation at > [http://impala.apache.org/docs/build/html/topics/impala_proxy.html#tut_proxy] > . > 2) Example configuration has following. > listen impala :25003 > mode tcp > option tcplog > balance leastconn > server symbolic_name_1 impala-host-1.example.com:21000 > server symbolic_name_2 impala-host-2.example.com:21000 > server symbolic_name_3 impala-host-3.example.com:21000 > server symbolic_name_4 impala-host-4.example.com:21000 > ** > Request :- > ** > There should be a "check" keyword at the end of each line that begins with > "server". Can we please add? (most users are likely to copy - paste and > replication and hence it would be best to keep it correct) > This is correct for the following pool mentioned later in the configuration. > > listen impalajdbc :21051 > mode tcp > option tcplog > balance source > server symbolic_name_5 impala-host-1.example.com:21050 check > server symbolic_name_6 impala-host-2.example.com:21050 check > server symbolic_name_7 impala-host-3.example.com:21050 check > server symbolic_name_8 impala-host-4.example.com:21050 check -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7173) [DOCS] Update load balancer configuration in our docs
Laurel Hale created IMPALA-7173: --- Summary: [DOCS] Update load balancer configuration in our docs Key: IMPALA-7173 URL: https://issues.apache.org/jira/browse/IMPALA-7173 Project: IMPALA Issue Type: Documentation Components: Docs Affects Versions: Impala 2.11.0 Reporter: Laurel Hale This Jira is filed on behalf of Sudarshan Sharma: 1) Please refer following documentation at [https://www.cloudera.com/documentation/enterprise/5-14-x/topics/impala_proxy.html#tut_proxy] . 2) Example configuration has following. listen impala :25003 mode tcp option tcplog balance leastconn server symbolic_name_1 impala-host-1.example.com:21000 server symbolic_name_2 impala-host-2.example.com:21000 server symbolic_name_3 impala-host-3.example.com:21000 server symbolic_name_4 impala-host-4.example.com:21000 ** Request :- ** There should be a "check" keyword at the end of each line that begins with "server". Can we please add? (most users are likely to copy - paste and replication and hence it would be best to keep it correct) This is correct for the following pool mentioned later in the configuration. listen impalajdbc :21051 mode tcp option tcplog balance source server symbolic_name_5 impala-host-1.example.com:21050 check server symbolic_name_6 impala-host-2.example.com:21050 check server symbolic_name_7 impala-host-3.example.com:21050 check server symbolic_name_8 impala-host-4.example.com:21050 check -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7172) Statestore should verify that all subscribers are running the same version of Impala
Mostafa Mokhtar created IMPALA-7172:
---
Summary: Statestore should verify that all subscribers are running
the same version of Impala
Key: IMPALA-7172
URL: https://issues.apache.org/jira/browse/IMPALA-7172
Project: IMPALA
Issue Type: New Feature
Components: Distributed Exec
Affects Versions: Impala 2.13.0
Reporter: Mostafa Mokhtar
While running a metadata test which uses sync_ddl=1, tests appeared to hang
indefinitely.
Turns out one of the Impala daemons was running an older build which caused
statestore topic updates to continuously fail.
Ideally the SS should track the version across subscribers and black list the
ones that don't match the SS and CS version.
Logs from SS
{code}
I0614 11:11:04.410529 57312 statestore.cc:259] Preparing initial
impala-membership topic update for impa...@vb0204.halxg.cloudera.com:22000.
Size = 2.06 KB
I0614 11:11:04.411222 57312 client-cache.cc:82] ReopenClient(): re-creating
client for vb0204.halxg.cloudera.com:23000
I0614 11:11:04.411821 57312 client-cache.h:304] RPC Error: Client for
vb0204.halxg.cloudera.com:23000 hit an unexpected exception: No more data to
read., type: N6apache6thrift9transport19TTransportExceptionE, rpc:
N6impala20TUpdateStateResponseE, send: done
I0614 11:11:04.411831 57312 client-cache.cc:174] Broken Connection, destroy
client for vb0204.halxg.cloudera.com:23000
I0614 11:11:04.411861 57312 statestore.cc:891] Unable to send priority topic
update message to subscriber impa...@vb0204.halxg.cloudera.com:22000, received
error: RPC Error: Client for vb0204.halxg.cloudera.com:23000 hit an unexpected
exception: No more data to read., type:
N6apache6thrift9transport19TTransportExceptionE, rpc:
N6impala20TUpdateStateResponseE, send: done
{code}
Log from Impalad
{code}
I0614 11:03:19.479164 41915 thrift-util.cc:123] TAcceptQueueServer exception:
N6apache6thrift8protocol18TProtocolExceptionE: TProtocolException: Invalid data
I0614 11:03:19.680028 41916 thrift-util.cc:123] TAcceptQueueServer exception:
N6apache6thrift8protocol18TProtocolExceptionE: TProtocolException: Invalid data
I0614 11:03:19.680776 41917 thrift-util.cc:123] TAcceptQueueServer exception:
N6apache6thrift8protocol18TProtocolExceptionE: TProtocolException: Invalid data
I0614 11:03:19.881295 41918 thrift-util.cc:123] TAcceptQueueServer exception:
N6apache6thrift8protocol18TProtocolExceptionE: TProtocolException: Invalid data
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7171) Add docs for Kudu insert partitioning/sorting
[ https://issues.apache.org/jira/browse/IMPALA-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7171: Component/s: Docs > Add docs for Kudu insert partitioning/sorting > - > > Key: IMPALA-7171 > URL: https://issues.apache.org/jira/browse/IMPALA-7171 > Project: IMPALA > Issue Type: Improvement > Components: Docs >Reporter: Thomas Tauber-Marshall >Assignee: Alex Rodoni >Priority: Major > Labels: docs > > On the page: > http://impala.apache.org/docs/build3x/html/topics/impala_kudu.html, at the > end of the section: "Impala DML Support for Kudu Tables (INSERT, UPDATE, > DELETE, UPSERT)", we should add text like: > Starting from Impala 2.9, Impala will automatically add a partition and sort > step to INSERTs before sending the rows to Kudu. Since Kudu partitions and > sorts rows on write, pre-partitioning and sorting takes some of the load off > of Kudu, and helps ensure that large INSERTs complete without timing out, but > it may slow down the end-to-end performance of the INSERT. Starting from > Impala 2.10, the hints "/* +noshuffle,noclustered */" may be used to turn > this pre-partitioning and sorting off. Additionally, since sorting may > consume a lot of memory, users should consider setting a "mem_limit" for > these queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7171) Add docs for Kudu insert partitioning/sorting
[ https://issues.apache.org/jira/browse/IMPALA-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni reassigned IMPALA-7171: --- Assignee: Alex Rodoni > Add docs for Kudu insert partitioning/sorting > - > > Key: IMPALA-7171 > URL: https://issues.apache.org/jira/browse/IMPALA-7171 > Project: IMPALA > Issue Type: Improvement > Components: Docs >Reporter: Thomas Tauber-Marshall >Assignee: Alex Rodoni >Priority: Major > Labels: docs > > On the page: > http://impala.apache.org/docs/build3x/html/topics/impala_kudu.html, at the > end of the section: "Impala DML Support for Kudu Tables (INSERT, UPDATE, > DELETE, UPSERT)", we should add text like: > Starting from Impala 2.9, Impala will automatically add a partition and sort > step to INSERTs before sending the rows to Kudu. Since Kudu partitions and > sorts rows on write, pre-partitioning and sorting takes some of the load off > of Kudu, and helps ensure that large INSERTs complete without timing out, but > it may slow down the end-to-end performance of the INSERT. Starting from > Impala 2.10, the hints "/* +noshuffle,noclustered */" may be used to turn > this pre-partitioning and sorting off. Additionally, since sorting may > consume a lot of memory, users should consider setting a "mem_limit" for > these queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7171) Add docs for Kudu insert partitioning/sorting
Thomas Tauber-Marshall created IMPALA-7171: -- Summary: Add docs for Kudu insert partitioning/sorting Key: IMPALA-7171 URL: https://issues.apache.org/jira/browse/IMPALA-7171 Project: IMPALA Issue Type: Improvement Reporter: Thomas Tauber-Marshall On the page: http://impala.apache.org/docs/build3x/html/topics/impala_kudu.html, at the end of the section: "Impala DML Support for Kudu Tables (INSERT, UPDATE, DELETE, UPSERT)", we should add text like: Starting from Impala 2.9, Impala will automatically add a partition and sort step to INSERTs before sending the rows to Kudu. Since Kudu partitions and sorts rows on write, pre-partitioning and sorting takes some of the load off of Kudu, and helps ensure that large INSERTs complete without timing out, but it may slow down the end-to-end performance of the INSERT. Starting from Impala 2.10, the hints "/* +noshuffle,noclustered */" may be used to turn this pre-partitioning and sorting off. Additionally, since sorting may consume a lot of memory, users should consider setting a "mem_limit" for these queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7151) session-expiry-test failed - unable to open ThriftServer port
[
https://issues.apache.org/jira/browse/IMPALA-7151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512778#comment-16512778
]
Tim Armstrong commented on IMPALA-7151:
---
I looked at the original patch for the ephemeral ports and Casey had a good
suggestion of just listening on port 0 and exposing the socket chosen.
https://gerrit.cloudera.org/#/c/1520/2/be/src/testutil/in-process-servers.cc@51
It looks like the thrift sockets support this:
[https://github.com/apache/thrift/blob/82ae9575cdc112088771fc7b876f75e1e4d85ebb/lib/cpp/test/TServerSocketTest.cpp#L36]
> session-expiry-test failed - unable to open ThriftServer port
> -
>
> Key: IMPALA-7151
> URL: https://issues.apache.org/jira/browse/IMPALA-7151
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: flaky
> Attachments: session-expiry-test.ERROR, session-expiry-test.FATAL,
> session-expiry-test.INFO, session-expiry-test.WARNING
>
>
> {noformat}
> ThriftServer 'backend' (on port: 52436) exited due to TException: Could not
> bind: Transport endpoint is not connected
> {noformat}
> This is similar to IMPALA-5499 but is on CentOS 7:
> {noformat}
> # lsb_release -a
> LSB Version:
> :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
> Distributor ID: CentOS
> Description: CentOS Linux release 7.4.1708 (Core)
> Release: 7.4.1708
> Codename: Core
> {noformat}
> It's in the ephemeral port range:
> {noformat}
> # cat /proc/sys/net/ipv4/ip_local_port_range
> 32768 60999
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-7165) Impala Doc: Example for Dynamic Partition Pruning need to be improved
[ https://issues.apache.org/jira/browse/IMPALA-7165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni closed IMPALA-7165. --- Resolution: Fixed Fix Version/s: Impala 3.1.0 Impala 2.13.0 > Impala Doc: Example for Dynamic Partition Pruning need to be improved > - > > Key: IMPALA-7165 > URL: https://issues.apache.org/jira/browse/IMPALA-7165 > Project: IMPALA > Issue Type: Bug > Components: Docs >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Fix For: Impala 2.13.0, Impala 3.1.0 > > > The documentation provided here: > [https://impala.apache.org/docs/build/html/topics/impala_partitioning.html#dynamic_partition_pruning] > for Impala's Dynamic Partition Pruning uses wrong example. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7147) Hit DCHECK in Parquet fuzz test
[
https://issues.apache.org/jira/browse/IMPALA-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csaba Ringhofer resolved IMPALA-7147.
-
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Hit DCHECK in Parquet fuzz test
> ---
>
> Key: IMPALA-7147
> URL: https://issues.apache.org/jira/browse/IMPALA-7147
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Csaba Ringhofer
>Priority: Critical
> Labels: crash
> Fix For: Impala 3.1.0
>
>
> This test was running:
> {noformat}
> 10:24:44 [gw3] ERROR
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[exec_option:
> {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5',
> 'abort_on_error': False, 'mem_limit': '512m', 'num_nodes': 0} | table_format:
> parquet/none]
> {noformat}
> {noformat}
> F0607 10:23:52.436451 9895 bit-stream-utils.inline.h:111] Check failed:
> bit_width <= sizeof(T) * 8 (9 vs. 8)
> {noformat}
> {noformat}
> CORE: ./core.1528392237.27349.impalad
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/build/late'.
> Program terminated with signal 6, Aborted.
> #0 0x7f30633008e5 in raise () from /lib64/libc.so.6
> To enable execution of this file add
> add-auto-load-safe-path
> /data/jenkins/workspace/impala-asf-master-core-local/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> To completely disable this security protection add
> set auto-load safe-path /
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> For more information about this security protection see the
> "Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
> info "(gdb)Auto-loading safe path"
> #0 0x7f30633008e5 in raise () from /lib64/libc.so.6
> #1 0x7f30633020c5 in abort () from /lib64/libc.so.6
> #2 0x042bdf64 in google::DumpStackTraceAndExit() ()
> #3 0x042b49dd in google::LogMessage::Fail() ()
> #4 0x042b6282 in google::LogMessage::SendToLog() ()
> #5 0x042b43b7 in google::LogMessage::Flush() ()
> #6 0x042b797e in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x022d001f in
> impala::BatchedBitReader::UnpackAndDecodeBatch (this=0xe8cc720,
> bit_width=9, dict=0xef5a720 "\a\b\t", dict_len=10, num_values=128,
> v=0xe8cc800 "") at
> /data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/util/bit-stream-utils.inline.h:111
> #8 0x022ccc56 in impala::RleBatchDecoder int>::DecodeLiteralValues (this=0xe8cc720,
> num_literals_to_consume=128, dict=0xef5a720 "\a\b\t", dict_len=10,
> values=0xe8cc800 "") at
> /data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/util/rle-encoding.h:569
> #9 0x022ca1a8 in impala::DictDecoder::DecodeNextValue
> (this=0xe8cc718, value=0x1cd00c057 "") at
> /data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/util/dict-encoding.h:469
> #10 0x022a8c3f in GetNextValue (this=0xe8cc480, max_values=1,
> tuple_size=90, tuple_mem=0x1cd00c000 "", num_values=0x7f2f9e5ad600) at
> /data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/util/dict-encoding.h:447
> #11 DecodeValue<(parquet::Encoding::type)2> (this=0xe8cc480,
> max_values=1, tuple_size=90, tuple_mem=0x1cd00c000 "",
> num_values=0x7f2f9e5ad600) at
> /data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/exec/parquet-column-readers.cc:625
> #12 ReadSlot<(parquet::Encoding::type)2, false> (this=0xe8cc480,
> max_values=1, tuple_size=90, tuple_mem=0x1cd00c000 "",
> num_values=0x7f2f9e5ad600) at
> /data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/exec/parquet-column-readers.cc:606
> #13 impala::ScalarColumnReader true>::MaterializeValueBatch
> (this=0xe8cc480, max_values=1, tuple_size=90, tuple_mem=0x1cd00c000 "",
> num_values=0x7f2f9e5ad600) at
> /data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/exec/parquet-column-readers.cc:555
> #14 0x022914df in impala::ScalarColumnReader (parquet::Type::type)1, true>::MaterializeValueBatch (this=0xe8cc480,
> max_values=1, tuple_size=90, tuple_mem=0x1cd00c000 "",
> num_values=0x7f2f9e5ad600) at
> /data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/exec/parquet-column-readers.cc:581
> #15 0x02278854 in impala::ScalarColumnReader (parquet::Type::type)1, true>::ReadValueBatch (this=0xe8cc480,
> max_values=1, tuple_size=90, tuple_mem=0x1cd00c000 "",
> num_values=0xab914b0) at
>
[jira] [Commented] (IMPALA-7147) Hit DCHECK in Parquet fuzz test
[
https://issues.apache.org/jira/browse/IMPALA-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512424#comment-16512424
]
Csaba Ringhofer commented on IMPALA-7147:
-
commit: 06fe3210509424cf50fdd3cc919466a1ce6bc842
IMPALA-7417: Remove DCHECKs with unnecessary constraint on dictionary encoding
bit width
Reading dictionary encoded Parquet data pages where the bit width is
larger than the encoded type's size (e.g. coding 8 bit TINYINT with
16 bit dictionary indices) led to DCHECK error in debug builds.
Impala does not create such parquet files (an N bit type can have
maximum 2^N distinct values, so N bit dictionary indices are enough
for a dictionary that contains every possible value), but the Parquet
standard does not forbid to do so.
These DCHECKs were probably introduced by a copy paste error (similar
checks exist in the non-dictionary encoded bit reader functions,
where they are valid).
Testing:
- a new test is added to check that these data pages can be decoded
correctly
Change-Id: I9ff3b00cbcab09dec11b3607d7d9a9c2c0025e1a
Reviewed-on: http://gerrit.cloudera.org:8080/10683
Reviewed-by: Tim Armstrong
Tested-by: Impala Public Jenkins
> Hit DCHECK in Parquet fuzz test
> ---
>
> Key: IMPALA-7147
> URL: https://issues.apache.org/jira/browse/IMPALA-7147
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Csaba Ringhofer
>Priority: Critical
> Labels: crash
> Fix For: Impala 3.1.0
>
>
> This test was running:
> {noformat}
> 10:24:44 [gw3] ERROR
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[exec_option:
> {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5',
> 'abort_on_error': False, 'mem_limit': '512m', 'num_nodes': 0} | table_format:
> parquet/none]
> {noformat}
> {noformat}
> F0607 10:23:52.436451 9895 bit-stream-utils.inline.h:111] Check failed:
> bit_width <= sizeof(T) * 8 (9 vs. 8)
> {noformat}
> {noformat}
> CORE: ./core.1528392237.27349.impalad
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/build/late'.
> Program terminated with signal 6, Aborted.
> #0 0x7f30633008e5 in raise () from /lib64/libc.so.6
> To enable execution of this file add
> add-auto-load-safe-path
> /data/jenkins/workspace/impala-asf-master-core-local/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> To completely disable this security protection add
> set auto-load safe-path /
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> For more information about this security protection see the
> "Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
> info "(gdb)Auto-loading safe path"
> #0 0x7f30633008e5 in raise () from /lib64/libc.so.6
> #1 0x7f30633020c5 in abort () from /lib64/libc.so.6
> #2 0x042bdf64 in google::DumpStackTraceAndExit() ()
> #3 0x042b49dd in google::LogMessage::Fail() ()
> #4 0x042b6282 in google::LogMessage::SendToLog() ()
> #5 0x042b43b7 in google::LogMessage::Flush() ()
> #6 0x042b797e in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x022d001f in
> impala::BatchedBitReader::UnpackAndDecodeBatch (this=0xe8cc720,
> bit_width=9, dict=0xef5a720 "\a\b\t", dict_len=10, num_values=128,
> v=0xe8cc800 "") at
> /data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/util/bit-stream-utils.inline.h:111
> #8 0x022ccc56 in impala::RleBatchDecoder int>::DecodeLiteralValues (this=0xe8cc720,
> num_literals_to_consume=128, dict=0xef5a720 "\a\b\t", dict_len=10,
> values=0xe8cc800 "") at
> /data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/util/rle-encoding.h:569
> #9 0x022ca1a8 in impala::DictDecoder::DecodeNextValue
> (this=0xe8cc718, value=0x1cd00c057 "") at
> /data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/util/dict-encoding.h:469
> #10 0x022a8c3f in GetNextValue (this=0xe8cc480, max_values=1,
> tuple_size=90, tuple_mem=0x1cd00c000 "", num_values=0x7f2f9e5ad600) at
> /data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/util/dict-encoding.h:447
> #11 DecodeValue<(parquet::Encoding::type)2> (this=0xe8cc480,
> max_values=1, tuple_size=90, tuple_mem=0x1cd00c000 "",
> num_values=0x7f2f9e5ad600) at
> /data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/exec/parquet-column-readers.cc:625
> #12 ReadSlot<(parquet::Encoding::type)2, false> (this=0xe8cc480,
> max_values=1, tuple_size=90, tuple_mem=0x1cd00c000 "",
>
