[jira] [Resolved] (IMPALA-8785) DEBUG build images should be tagged differently from release build
[ https://issues.apache.org/jira/browse/IMPALA-8785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-8785. --- Resolution: Fixed Fix Version/s: Impala 3.3.0 > DEBUG build images should be tagged differently from release build > -- > > Key: IMPALA-8785 > URL: https://issues.apache.org/jira/browse/IMPALA-8785 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Critical > Labels: docker > Fix For: Impala 3.3.0 > > > Currently the container build gives debug and release images the same tag > names. This is fine for development, since you'll generally be working on one > build at a time, but is confusing if you need to integrate this into a more > general build process, since debug and release images are really separate > artifacts. > One example I ran into is a case when I ran ./buildall.sh ... > -release_and_debug, then wanted to build docker images for the release > artifacts. This is not currently possible in a sane way. > I think we should tag images with a -debug suffix if they were generated from > debug artifacts. start-impala-cluster would need to be smart about picking > the image to use. We may want to have different build targets to build the > different flavours of image, too. > Note that there are many flavours of debug images, e.g. ASAN, but I think > simply separating release and debug would avoid a lot of confusion. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8816) custom cluster tests in precommit are taking close to 2 hours
[ https://issues.apache.org/jira/browse/IMPALA-8816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896672#comment-16896672 ] Tim Armstrong commented on IMPALA-8816: --- Actually, even without a special mode for start-impala-cluster, we can have detect the failure by polling that the processes still exist. I prototyped it like this: {noformat} tarmstrong@tarmstrong-box:~/Impala/incubator-impala$ git show commit ff30f358118e392bf0912565b31cf4137f71a656 Author: Tim Armstrong Date: Tue Jul 30 18:37:41 2019 -0700 Prototype fast fail diff --git a/be/src/service/impala-server.cc b/be/src/service/impala-server.cc index d2f44d4..2a983e7 100644 --- a/be/src/service/impala-server.cc +++ b/be/src/service/impala-server.cc @@ -2495,6 +2495,9 @@ Status ImpalaServer::Start(int32_t thrift_be_port, int32_t beeswax_port, int32_t // then wait for the initial catalog update. RETURN_IF_ERROR(exec_env_->StartStatestoreSubscriberService()); + SleepForMs(1); + exit(1); + if (FLAGS_is_coordinator) exec_env_->frontend()->WaitForCatalog(); SSLProtocol ssl_version = SSLProtocol::TLSv1_0; diff --git a/tests/common/impala_cluster.py b/tests/common/impala_cluster.py index 6136091..fb900ea 100644 --- a/tests/common/impala_cluster.py +++ b/tests/common/impala_cluster.py @@ -169,9 +169,18 @@ class ImpalaCluster(object): if expected_num_ready_impalads is None: expected_num_ready_impalads = len(self.impalads) +def impalads_went_away(): + print "Check" + self.refresh() + # Number of impalads should not change. + assert expected_num_impalads == len(self.impalads) + assert self.statestored is not None + assert self.catalogd is not None + + for impalad in self.impalads: impalad.service.wait_for_num_known_live_backends(expected_num_ready_impalads, - timeout=CLUSTER_WAIT_TIMEOUT_IN_SECONDS, interval=2) + timeout=CLUSTER_WAIT_TIMEOUT_IN_SECONDS, interval=2, early_term_fn=impalads_went_away) if (impalad._get_arg_value("is_coordinator", default="true") == "true" and impalad._get_arg_value("stress_catalog_init_delay_ms", default=0) == 0): impalad.wait_for_catalog() diff --git a/tests/common/impala_service.py b/tests/common/impala_service.py index cba0a0f..7b06f3d 100644 --- a/tests/common/impala_service.py +++ b/tests/common/impala_service.py @@ -251,9 +251,10 @@ class ImpaladService(BaseImpalaService): return False def wait_for_num_known_live_backends(self, expected_value, timeout=30, interval=1, - include_shutting_down=True): + include_shutting_down=True, early_term_fn=lambda: False): start_time = time() while (time() - start_time < timeout): + early_term_fn() value = None try: value = self.get_num_known_live_backends(timeout=timeout, interval=interval, {noformat} > custom cluster tests in precommit are taking close to 2 hours > - > > Key: IMPALA-8816 > URL: https://issues.apache.org/jira/browse/IMPALA-8816 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > > This is affecting precommit times substantially. We should either speed up > the tests or, more likely, move some to exhaustive. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8816) custom cluster tests in precommit are taking close to 2 hours
[ https://issues.apache.org/jira/browse/IMPALA-8816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896665#comment-16896665 ] Tim Armstrong commented on IMPALA-8816: --- A large part of the problem is that start-impala-cluster.py is not designed to support these kind of startup failure tests. Some startup failure tests end up waiting for CLUSTER_WAIT_TIMEOUT_IN_SECONDS for a metric to increment. This is non-deterministic. It seems like much of the time the tests fail with a shorter timeout when processes fail to appear here - https://github.com/apache/impala/blob/72c9370856d7436885adbee3e8da7e7d9336df15/tests/common/impala_cluster.py#L166. But unlucky ones get stuck waiting for the longer metric timeout. {noformat} custom_cluster.test_query_event_hooks.TestHooksStartupFail.test_hook_startup_fail (from pytest) Took 4 min 14 sec. add description Standard Error -- 2019-07-30 22:53:23,755 INFO MainThread: Starting cluster with command: /home/ubuntu/Impala/bin/start-impala-cluster.py --cluster_size=1 --num_coordinators=1 --log_dir=/home/ubuntu/Impala/logs/custom_cluster_tests/test_hooks_startup_fail_uLTriL --log_level=1 '--impalad_args=--query_event_hook_classes=org.apache.impala.testutil.AlwaysErrorQueryEventHook --minidump_path=/tmp/tmpBDmylN ' '--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 ...[truncated 14624 chars]... ome/ubuntu/Impala/tests/common/impala_cluster.py", line 174, in wait_until_ready timeout=CLUSTER_WAIT_TIMEOUT_IN_SECONDS, interval=2) File "/home/ubuntu/Impala/tests/common/impala_service.py", line 270, in wait_for_num_known_live_backends assert 0, 'num_known_live_backends did not reach expected value in time' AssertionError: num_known_live_backends did not reach expected value in time -- 2019-07-30 22:57:29,083 DEBUGMainThread: Found 0 impalad/1 statestored/1 catalogd process(es) {noformat} I think we could fix this entirely by specialising start-impala-cluster to have an "expected startup failure mode" that waits for the number of impalads to hit 0, which should be quick, then throws an exception. > custom cluster tests in precommit are taking close to 2 hours > - > > Key: IMPALA-8816 > URL: https://issues.apache.org/jira/browse/IMPALA-8816 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > > This is affecting precommit times substantially. We should either speed up > the tests or, more likely, move some to exhaustive. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8668) Document impala-shell HS2 support
[ https://issues.apache.org/jira/browse/IMPALA-8668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8668 started by Alex Rodoni. --- > Document impala-shell HS2 support > - > > Key: IMPALA-8668 > URL: https://issues.apache.org/jira/browse/IMPALA-8668 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Tim Armstrong >Assignee: Alex Rodoni >Priority: Major > Labels: docs, future_release_doc, in_33 > > https://gerrit.cloudera.org/#/c/13961/ -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8668) Document impala-shell HS2 support
[ https://issues.apache.org/jira/browse/IMPALA-8668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-8668: Description: https://gerrit.cloudera.org/#/c/13961/ > Document impala-shell HS2 support > - > > Key: IMPALA-8668 > URL: https://issues.apache.org/jira/browse/IMPALA-8668 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Tim Armstrong >Assignee: Alex Rodoni >Priority: Major > Labels: docs, future_release_doc, in_33 > > https://gerrit.cloudera.org/#/c/13961/ -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8791) Handle the case where there is no fragment scheduled on the coordinator for a query
[ https://issues.apache.org/jira/browse/IMPALA-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikramjeet Vig updated IMPALA-8791: --- Description: For insert statements executed on a dedicated coord, the fragments get schduled only on executors but a query state object still gets started up on the coord host with the coord_mem_limit. we end up with a situation where the mem admitted is zero for the coord but the mem_reserved is non-zero which would affect other admission decisions. There is also a case where there is no coordinator fragment but the execution fragment gets scheduled on the coord (eg. insert into values) for this case, the mem admitted is per_backend_mem_limit_ but the mem limit applied to the coord query state is coord_backend_mem_limit_ which again causes a inconsistency. was:For insert statements executed on a dedicated coord, the fragments get schduled only on executors but a query state object still gets started up on the coord host with the coord_mem_limit. we end up with a situation where the mem admitted is zero for the coord but the mem_reserved is non-zero which would affect other admission decisions. > Handle the case where there is no fragment scheduled on the coordinator for a > query > --- > > Key: IMPALA-8791 > URL: https://issues.apache.org/jira/browse/IMPALA-8791 > Project: IMPALA > Issue Type: Bug >Affects Versions: Product Backlog >Reporter: Bikramjeet Vig >Assignee: Bikramjeet Vig >Priority: Major > > For insert statements executed on a dedicated coord, the fragments get > schduled only on executors but a query state object still gets started up on > the coord host with the coord_mem_limit. we end up with a situation where the > mem admitted is zero for the coord but the mem_reserved is non-zero which > would affect other admission decisions. > There is also a case where there is no coordinator fragment but the execution > fragment gets scheduled on the coord (eg. insert into values) for > this case, the mem admitted is per_backend_mem_limit_ but the mem limit > applied to the coord query state is coord_backend_mem_limit_ which again > causes a inconsistency. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8456) Impala Doc: Document HTTP based HS2/beeswax endpoints on coordinators
[ https://issues.apache.org/jira/browse/IMPALA-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896639#comment-16896639 ] Alex Rodoni commented on IMPALA-8456: - https://gerrit.cloudera.org/#/c/13960/ > Impala Doc: Document HTTP based HS2/beeswax endpoints on coordinators > - > > Key: IMPALA-8456 > URL: https://issues.apache.org/jira/browse/IMPALA-8456 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > > Also document IMPALA-8717 > https://issues.apache.org/jira/browse/IMPALA-8783 -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8456) Impala Doc: Document HTTP based HS2/beeswax endpoints on coordinators
[ https://issues.apache.org/jira/browse/IMPALA-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8456 started by Alex Rodoni. --- > Impala Doc: Document HTTP based HS2/beeswax endpoints on coordinators > - > > Key: IMPALA-8456 > URL: https://issues.apache.org/jira/browse/IMPALA-8456 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > > Also document IMPALA-8717 > https://issues.apache.org/jira/browse/IMPALA-8783 -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8816) custom cluster tests in precommit are taking close to 2 hours
Tim Armstrong created IMPALA-8816: - Summary: custom cluster tests in precommit are taking close to 2 hours Key: IMPALA-8816 URL: https://issues.apache.org/jira/browse/IMPALA-8816 Project: IMPALA Issue Type: Bug Components: Infrastructure Affects Versions: Impala 3.3.0 Reporter: Tim Armstrong Assignee: Tim Armstrong This is affecting precommit times substantially. We should either speed up the tests or, more likely, move some to exhaustive. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8815) Ranger fail to start in minicluster if you source set-classpath.sh
[ https://issues.apache.org/jira/browse/IMPALA-8815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896625#comment-16896625 ] Tim Armstrong commented on IMPALA-8815: --- After I fixed the immediate failure, there were some more failures that I think were because of incompatible versions of dependencies - it was unable to find some methods in some jackson class. > Ranger fail to start in minicluster if you source set-classpath.sh > -- > > Key: IMPALA-8815 > URL: https://issues.apache.org/jira/browse/IMPALA-8815 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Critical > > The below appear in logs/cluster/ranger/catalina.out > {noformat} > java.io.FileNotFoundException: > /home/tarmstrong/Impala/incubator-impala/toolchain/cdh_components-1173663/hive-2.1.1-cdh6.x-SNAPSHOT/lib/datanucleus-api-jdo-3.2.1.jar > (No such file or directory) > at java.util.zip.ZipFile.open(Native Method) > at java.util.zip.ZipFile.(ZipFile.java:225) > at java.util.zip.ZipFile.(ZipFile.java:155) > at java.util.jar.JarFile.(JarFile.java:166) > at java.util.jar.JarFile.(JarFile.java:103) > at sun.net.www.protocol.jar.URLJarFile.(URLJarFile.java:93) > at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69) > at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99) > at > sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122) > at > sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89) > at org.apache.tomcat.util.scan.FileUrlJar.(FileUrlJar.java:48) > at > org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:34) > at > org.apache.catalina.startup.ContextConfig$FragmentJarScannerCallback.scan(ContextConfig.java:2682) > at > org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:325) > at > org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:235) > at > org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:196) > at > org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1913) > at > org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1264) > at > org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:881) > at > org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:388) > at > org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117) > at > org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90) > at > org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5606) > at > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) > at > org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1707) > at > org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1697) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8785) DEBUG build images should be tagged differently from release build
[ https://issues.apache.org/jira/browse/IMPALA-8785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896626#comment-16896626 ] ASF subversion and git services commented on IMPALA-8785: - Commit def70c241dffcdab8301cbc7456ea6680ca01692 in impala's branch refs/heads/master from Tim Armstrong [ https://gitbox.apache.org/repos/asf?p=impala.git;h=def70c2 ] IMPALA-8785: give debug docker images a different name * Build scripts are generalised to have different targets for release and debug images. * Added new targets for the debug images: docker_debug_images, statestored_debug images. The release images still have the same names. * Separate build contexts are set up for the different base images. * The debug or release base image can be specified as the FROM for the daemon images. * start-impala-cluster.py picks the correct images for the build type Future work: We would like to generalise this to allow building from non-ubuntu-16.04 base images. This probably requires another layer of dockerfiles to specify a base image for impala_base with the required packages installed. Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 Reviewed-on: http://gerrit.cloudera.org:8080/13905 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins > DEBUG build images should be tagged differently from release build > -- > > Key: IMPALA-8785 > URL: https://issues.apache.org/jira/browse/IMPALA-8785 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Critical > Labels: docker > > Currently the container build gives debug and release images the same tag > names. This is fine for development, since you'll generally be working on one > build at a time, but is confusing if you need to integrate this into a more > general build process, since debug and release images are really separate > artifacts. > One example I ran into is a case when I ran ./buildall.sh ... > -release_and_debug, then wanted to build docker images for the release > artifacts. This is not currently possible in a sane way. > I think we should tag images with a -debug suffix if they were generated from > debug artifacts. start-impala-cluster would need to be smart about picking > the image to use. We may want to have different build targets to build the > different flavours of image, too. > Note that there are many flavours of debug images, e.g. ASAN, but I think > simply separating release and debug would avoid a lot of confusion. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8815) Ranger fail to start in minicluster if you source set-classpath.sh
Tim Armstrong created IMPALA-8815: - Summary: Ranger fail to start in minicluster if you source set-classpath.sh Key: IMPALA-8815 URL: https://issues.apache.org/jira/browse/IMPALA-8815 Project: IMPALA Issue Type: Bug Components: Infrastructure Reporter: Tim Armstrong Assignee: Tim Armstrong The below appear in logs/cluster/ranger/catalina.out {noformat} java.io.FileNotFoundException: /home/tarmstrong/Impala/incubator-impala/toolchain/cdh_components-1173663/hive-2.1.1-cdh6.x-SNAPSHOT/lib/datanucleus-api-jdo-3.2.1.jar (No such file or directory) at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.(ZipFile.java:225) at java.util.zip.ZipFile.(ZipFile.java:155) at java.util.jar.JarFile.(JarFile.java:166) at java.util.jar.JarFile.(JarFile.java:103) at sun.net.www.protocol.jar.URLJarFile.(URLJarFile.java:93) at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69) at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99) at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122) at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89) at org.apache.tomcat.util.scan.FileUrlJar.(FileUrlJar.java:48) at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:34) at org.apache.catalina.startup.ContextConfig$FragmentJarScannerCallback.scan(ContextConfig.java:2682) at org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:325) at org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:235) at org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:196) at org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1913) at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1264) at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:881) at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:388) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5606) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1707) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1697) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8814) Impala Doc: Webui and SPNEGO
[ https://issues.apache.org/jira/browse/IMPALA-8814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8814 started by Alex Rodoni. --- > Impala Doc: Webui and SPNEGO > > > Key: IMPALA-8814 > URL: https://issues.apache.org/jira/browse/IMPALA-8814 > Project: IMPALA > Issue Type: Task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > > https://gerrit.cloudera.org/#/c/13774/ > Doc review at: https://gerrit.cloudera.org/#/c/13958/ -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8814) Impala Doc: Webui and SPNEGO
[ https://issues.apache.org/jira/browse/IMPALA-8814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-8814: Description: https://gerrit.cloudera.org/#/c/13774/ Doc review at: https://gerrit.cloudera.org/#/c/13958/ was:https://gerrit.cloudera.org/#/c/13774/ > Impala Doc: Webui and SPNEGO > > > Key: IMPALA-8814 > URL: https://issues.apache.org/jira/browse/IMPALA-8814 > Project: IMPALA > Issue Type: Task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > > https://gerrit.cloudera.org/#/c/13774/ > Doc review at: https://gerrit.cloudera.org/#/c/13958/ -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8738) Add a column representing the type(table or view) in the show tables output
[ https://issues.apache.org/jira/browse/IMPALA-8738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896561#comment-16896561 ] Thomas Tauber-Marshall commented on IMPALA-8738: Seems like a reasonable suggestion. Do you know if other DBs, eg. Hive, do something similar? We tend to try to stick to general conventions with this sort of stuff > Add a column representing the type(table or view) in the show tables output > --- > > Key: IMPALA-8738 > URL: https://issues.apache.org/jira/browse/IMPALA-8738 > Project: IMPALA > Issue Type: Improvement > Components: Catalog, Frontend >Reporter: baotuquan >Assignee: baotuquan >Priority: Minor > Labels: features > > Now the output of the +*show tables*+ command in the system is as follows: > {code:java} > default> show tables; > Query: show tables > +--+ > | name | > +--+ > | table1 | > | view1 | > +--+ > {code} > I think we should add a column for the representation type,The output should > be like this > {code:java} > default> show tables; > Query: show tables > +---+ > | name | type | > ++ > | table1 | table | > | view1 | view | > ++ > {code} > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8814) Impala Doc: Webui and SPNEGO
Alex Rodoni created IMPALA-8814: --- Summary: Impala Doc: Webui and SPNEGO Key: IMPALA-8814 URL: https://issues.apache.org/jira/browse/IMPALA-8814 Project: IMPALA Issue Type: Task Components: Docs Reporter: Alex Rodoni Assignee: Alex Rodoni https://gerrit.cloudera.org/#/c/13774/ -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8813) Impala Doc: Read/Write Hive ACID tables
Alex Rodoni created IMPALA-8813: --- Summary: Impala Doc: Read/Write Hive ACID tables Key: IMPALA-8813 URL: https://issues.apache.org/jira/browse/IMPALA-8813 Project: IMPALA Issue Type: Sub-task Components: Docs Reporter: Alex Rodoni Assignee: Alex Rodoni -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8347) Support read/write Hive ACID tables
[ https://issues.apache.org/jira/browse/IMPALA-8347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni reassigned IMPALA-8347: --- Assignee: (was: Alex Rodoni) > Support read/write Hive ACID tables > --- > > Key: IMPALA-8347 > URL: https://issues.apache.org/jira/browse/IMPALA-8347 > Project: IMPALA > Issue Type: Epic > Components: Backend, Frontend >Affects Versions: Impala 3.2.0 >Reporter: Lars Volker >Priority: Critical > Labels: impala-acid > > Hive added ACID transactional support in HIVE-5317. Impala should be able to > read Hive ACID tables. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8347) Support read/write Hive ACID tables
[ https://issues.apache.org/jira/browse/IMPALA-8347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8347 started by Alex Rodoni. --- > Support read/write Hive ACID tables > --- > > Key: IMPALA-8347 > URL: https://issues.apache.org/jira/browse/IMPALA-8347 > Project: IMPALA > Issue Type: Epic > Components: Backend, Frontend >Affects Versions: Impala 3.2.0 >Reporter: Lars Volker >Assignee: Alex Rodoni >Priority: Critical > Labels: impala-acid > > Hive added ACID transactional support in HIVE-5317. Impala should be able to > read Hive ACID tables. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work stopped] (IMPALA-8347) Support read/write Hive ACID tables
[ https://issues.apache.org/jira/browse/IMPALA-8347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8347 stopped by Alex Rodoni. --- > Support read/write Hive ACID tables > --- > > Key: IMPALA-8347 > URL: https://issues.apache.org/jira/browse/IMPALA-8347 > Project: IMPALA > Issue Type: Epic > Components: Backend, Frontend >Affects Versions: Impala 3.2.0 >Reporter: Lars Volker >Assignee: Alex Rodoni >Priority: Critical > Labels: impala-acid > > Hive added ACID transactional support in HIVE-5317. Impala should be able to > read Hive ACID tables. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8636) Implement INSERT for insert-only ACID tables
[ https://issues.apache.org/jira/browse/IMPALA-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896484#comment-16896484 ] Alex Rodoni commented on IMPALA-8636: - [~boroknagyz] Any user-facing doc impact? > Implement INSERT for insert-only ACID tables > > > Key: IMPALA-8636 > URL: https://issues.apache.org/jira/browse/IMPALA-8636 > Project: IMPALA > Issue Type: New Feature >Reporter: Zoltán Borók-Nagy >Assignee: Zoltán Borók-Nagy >Priority: Critical > Labels: impala-acid > Fix For: Impala 4.0, Impala 3.3.0 > > > Impala should support insertion for insert-only ACID tables. > For this we need to allocate a write ID for the target table, and write the > data into the base/delta directories. > INSERT operation should create a new delta directory with the allocated write > ID. > INSERT OVERWRITE should create a new base directory with the allocated write > ID. This new base directory will only contain the data coming from this > operation. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8812) Impala Doc: SPLIT_PART to support negative indexes
Alex Rodoni created IMPALA-8812: --- Summary: Impala Doc: SPLIT_PART to support negative indexes Key: IMPALA-8812 URL: https://issues.apache.org/jira/browse/IMPALA-8812 Project: IMPALA Issue Type: Sub-task Components: Docs Reporter: Alex Rodoni Assignee: Alex Rodoni -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7770) SPLIT_PART to support negative indexes
[ https://issues.apache.org/jira/browse/IMPALA-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896456#comment-16896456 ] Alex Rodoni commented on IMPALA-7770: - [~norbertluksa] Are you targeting the 3.3 release for this feature? > SPLIT_PART to support negative indexes > -- > > Key: IMPALA-7770 > URL: https://issues.apache.org/jira/browse/IMPALA-7770 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tristan Stevens >Assignee: Norbert Luksa >Priority: Major > Labels: built-in-function > > Request is for SPLIT_PART to support negative indexes i.e. support right to > left searching. > See Snowflake documentation for details: > [https://docs.snowflake.net/manuals/sql-reference/functions/split_part.html|https://docs.snowflake.net/manuals/sql-reference/functions/split_part.html] > partNr: Requested part of the split (1-based). 0 is treated as 1. If the > value is negative, the parts are counted from the right side of the string. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8811) Impala Doc: query option to change default ACID type of new tables
Alex Rodoni created IMPALA-8811: --- Summary: Impala Doc: query option to change default ACID type of new tables Key: IMPALA-8811 URL: https://issues.apache.org/jira/browse/IMPALA-8811 Project: IMPALA Issue Type: Sub-task Components: Docs Reporter: Alex Rodoni Assignee: Alex Rodoni -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8808) Add query option to change default ACID type of new tables
[ https://issues.apache.org/jira/browse/IMPALA-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896438#comment-16896438 ] Alex Rodoni commented on IMPALA-8808: - [~csringhofer] Are you targeting this feature for 3.3? > Add query option to change default ACID type of new tables > -- > > Key: IMPALA-8808 > URL: https://issues.apache.org/jira/browse/IMPALA-8808 > Project: IMPALA > Issue Type: New Feature > Components: Frontend >Affects Versions: Impala 3.3.0 >Reporter: Csaba Ringhofer >Assignee: Csaba Ringhofer >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8810) ExplainTest.testScanNodeFsScheme fails if Orc support is disabled
Csaba Ringhofer created IMPALA-8810: --- Summary: ExplainTest.testScanNodeFsScheme fails if Orc support is disabled Key: IMPALA-8810 URL: https://issues.apache.org/jira/browse/IMPALA-8810 Project: IMPALA Issue Type: Bug Components: Frontend Affects Versions: Impala 3.3.0 Reporter: Csaba Ringhofer The problem is caused by going into this block when Orc support is disabled: https://github.com/apache/impala/blob/51f30a6b9088cf3a0217a49a5792f1b41a8ad51b/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java#L416 + that ExplainTest only mocks HdfsPartition.getFileFormat() but not HdfsPartition.part.getInputFormatDescriptor(). -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8809) Refresh a subset of partitions for ACID tables
Gabor Kaszab created IMPALA-8809: Summary: Refresh a subset of partitions for ACID tables Key: IMPALA-8809 URL: https://issues.apache.org/jira/browse/IMPALA-8809 Project: IMPALA Issue Type: Improvement Affects Versions: Impala 3.3.0 Reporter: Gabor Kaszab Enhancing REFRESH logic to handle ACID tables was covered by this change: https://issues.apache.org/jira/browse/IMPALA-8600 Basically each user initiated REFRESH PARTITION is rejected meanwhile the REFRESH_PARTITION event in event processor are actually doing a full table load for ACID tables. There is room for improvement: When a full table refresh is being executed on an ACID table we can have 2 scenarios: - If there was some schema changes then reload the full table. Identify such a scenario should be possible by checking the table-level writeId. However, there is a bug in Hive that it doesn't update that field for partitioned tables (https://issues.apache.org/jira/browse/HIVE-22062). This would be the desired way but could also be workarounded by checking other fields lik lastDdlChanged or such. - If a full table refresh is not needed then we should fetch the partition-level writeIds and reload only the ones that are out-of-date locally. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8339) Coordinator should be more resilient to fragment instances startup failure
[ https://issues.apache.org/jira/browse/IMPALA-8339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896223#comment-16896223 ] ASF subversion and git services commented on IMPALA-8339: - Commit dfc968dff1eabc71871e2d941fbe9539944c3e88 in impala's branch refs/heads/master from Thomas Tauber-Marshall [ https://gitbox.apache.org/repos/asf?p=impala.git;h=dfc968d ] IMPALA-8339: Add local executor blacklist to coordinators This patch adds the concept of a blacklist of executors to the coordinator, which removes executors from consideration for query scheduling. Blacklisting decisions are local to a given coordinator and are not included in statestore updates. The intention is to allow coordinators to be more aggressive about deciding that an exeutor is unhealthy or unavailable, to minimize failed queries in environments where cluster membership may be more variable, rather than having to wait on the statestore heartbeat mechanism to decide that the executor is down. For the first patch, executors will only be blacklisted if the KRPC status for Exec() is an error. Followup work will add blacklisting of executors in more complex scenarios, eg. if an executor appears to be a straggler. When a query is scheduled and there is currently some blacklisted executors, a new line 'Blacklisted Executors:' is added to the profile listing the hostnames of all such executors. Testing: - Added a case to the cluster mgr BE unit test that uses blacklisting. - Added e2e test cases for killing and restarting an impalad. - Manual randomized testing locally with iptables. TODO - Add an e2e test case where an impalad becomes briefly unreachable. - Manual/stress tests on a real cluster. Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Reviewed-on: http://gerrit.cloudera.org:8080/13868 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Coordinator should be more resilient to fragment instances startup failure > -- > > Key: IMPALA-8339 > URL: https://issues.apache.org/jira/browse/IMPALA-8339 > Project: IMPALA > Issue Type: Improvement > Components: Distributed Exec >Reporter: Michael Ho >Assignee: Thomas Tauber-Marshall >Priority: Critical > Labels: Availability, resilience > > Impala currently relies on statestore for cluster membership. When an Impala > executor goes offline, it may take a while for statestore to declare that > node as unavailable and for that information to be propagated to all > coordinator nodes. Within this window, some coordinator nodes may still > attempt to issue RPCs to the faulty node, resulting in RPC failures which > resulted in query failures. In other words, many queries may fail to start > within this window until all coordinator nodes get the latest information on > cluster membership. > Going forward, coordinator may need to fall back to using backup executors > for each fragments in case some of the executors are not available. Moreover, > *coordinator should treat the cluster membership information from statestore > (or any external source of truth e.g. etcd) as hints instead of ground truth* > and adjust the scheduling of fragment instances based on the availability of > the executors from the coordinator's perspective. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8808) Add query option to change default ACID type of new tables
[ https://issues.apache.org/jira/browse/IMPALA-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Csaba Ringhofer reassigned IMPALA-8808: --- Assignee: Csaba Ringhofer > Add query option to change default ACID type of new tables > -- > > Key: IMPALA-8808 > URL: https://issues.apache.org/jira/browse/IMPALA-8808 > Project: IMPALA > Issue Type: New Feature > Components: Frontend >Affects Versions: Impala 3.3.0 >Reporter: Csaba Ringhofer >Assignee: Csaba Ringhofer >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8808) Add query option to change default ACID type of new tables
Csaba Ringhofer created IMPALA-8808: --- Summary: Add query option to change default ACID type of new tables Key: IMPALA-8808 URL: https://issues.apache.org/jira/browse/IMPALA-8808 Project: IMPALA Issue Type: New Feature Components: Frontend Affects Versions: Impala 3.3.0 Reporter: Csaba Ringhofer -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8600) Reload partition does not work for transactional tables
[ https://issues.apache.org/jira/browse/IMPALA-8600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896178#comment-16896178 ] Gabor Kaszab commented on IMPALA-8600: -- I ran into a Hive issue during the implementation: Hive doesn't increment any writeIds if I change the schema of a partitioned ACID table. It increments the high watermark, however. I opened a Jira: https://issues.apache.org/jira/browse/HIVE-22062 > Reload partition does not work for transactional tables > --- > > Key: IMPALA-8600 > URL: https://issues.apache.org/jira/browse/IMPALA-8600 > Project: IMPALA > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Gabor Kaszab >Priority: Major > Labels: impala-acid > > If a table is transactional, a reload partition call should fetch the valid > writeIds. Without doing this, the reload will skip adding all the newly > created delta files of the transactional table pertaining to the new writeIds. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7770) SPLIT_PART to support negative indexes
[ https://issues.apache.org/jira/browse/IMPALA-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896104#comment-16896104 ] Norbert Luksa commented on IMPALA-7770: --- Feel free to review: https://gerrit.cloudera.org/#/c/13880/ > SPLIT_PART to support negative indexes > -- > > Key: IMPALA-7770 > URL: https://issues.apache.org/jira/browse/IMPALA-7770 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tristan Stevens >Assignee: Norbert Luksa >Priority: Major > Labels: built-in-function > > Request is for SPLIT_PART to support negative indexes i.e. support right to > left searching. > See Snowflake documentation for details: > [https://docs.snowflake.net/manuals/sql-reference/functions/split_part.html|https://docs.snowflake.net/manuals/sql-reference/functions/split_part.html] > partNr: Requested part of the split (1-based). 0 is treated as 1. If the > value is negative, the parts are counted from the right side of the string. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8752) Add Jaro-winkler edit distance and similarity built-in function
[ https://issues.apache.org/jira/browse/IMPALA-8752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896106#comment-16896106 ] Norbert Luksa commented on IMPALA-8752: --- https://gerrit.cloudera.org/#/c/13870/ > Add Jaro-winkler edit distance and similarity built-in function > --- > > Key: IMPALA-8752 > URL: https://issues.apache.org/jira/browse/IMPALA-8752 > Project: IMPALA > Issue Type: New Feature >Reporter: Norbert Luksa >Assignee: Norbert Luksa >Priority: Major > Labels: built-in-function > > References: > * [Apache commons - JaroWinklerDistance > |[https://commons.apache.org/proper/commons-text/apidocs/org/apache/commons/text/similarity/JaroWinklerDistance.html]] > * [Apache commons - JaroWinklerSimilarity > |[https://commons.apache.org/proper/commons-text/apidocs/org/apache/commons/text/similarity/JaroWinklerSimilarity.html]] > * [Oracle - > JARO_WINKLER[_SIMILARITY]|[https://oracle-base.com/articles/11g/utl_match-string-matching-in-oracle]] > Notable difference: > * With similarity, the Oracle version returns a normalized result ranging > from 0 to 100. > * In the Appache version, null values result in exceptions. > * Apache rounds the values to two digits > The scaling factor of the algorithm can be added as an extra/default argument. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org