[jira] [Updated] (IMPALA-8593) Prohibit write to bucketed table
[ https://issues.apache.org/jira/browse/IMPALA-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated IMPALA-8593: - Priority: Critical (was: Major) > Prohibit write to bucketed table > > > Key: IMPALA-8593 > URL: https://issues.apache.org/jira/browse/IMPALA-8593 > Project: IMPALA > Issue Type: New Feature > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Critical > Labels: impala-acid > > Impala does not support writing to bucketed tables, we need prohibit these > unsupported operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8593) Prohibit write to bucketed table
[ https://issues.apache.org/jira/browse/IMPALA-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated IMPALA-8593: - Epic Color: ghx-label-7 (was: ghx-label-2) > Prohibit write to bucketed table > > > Key: IMPALA-8593 > URL: https://issues.apache.org/jira/browse/IMPALA-8593 > Project: IMPALA > Issue Type: New Feature > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Labels: impala-acid > > Impala does not support writing to bucketed tables, we need prohibit these > unsupported operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8593) Prohibit write to bucketed table
[ https://issues.apache.org/jira/browse/IMPALA-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated IMPALA-8593: - Component/s: Frontend > Prohibit write to bucketed table > > > Key: IMPALA-8593 > URL: https://issues.apache.org/jira/browse/IMPALA-8593 > Project: IMPALA > Issue Type: New Feature > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > > Impala does not support writing to bucketed tables, we need prohibit these > unsupported operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8593) Prohibit write to bucketed table
[ https://issues.apache.org/jira/browse/IMPALA-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated IMPALA-8593: - Labels: impala-acid (was: ) > Prohibit write to bucketed table > > > Key: IMPALA-8593 > URL: https://issues.apache.org/jira/browse/IMPALA-8593 > Project: IMPALA > Issue Type: New Feature > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Labels: impala-acid > > Impala does not support writing to bucketed tables, we need prohibit these > unsupported operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8593) Prohibit write to bucketed table
[ https://issues.apache.org/jira/browse/IMPALA-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated IMPALA-8593: - Affects Version/s: Impala 3.1.0 > Prohibit write to bucketed table > > > Key: IMPALA-8593 > URL: https://issues.apache.org/jira/browse/IMPALA-8593 > Project: IMPALA > Issue Type: New Feature >Affects Versions: Impala 3.1.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > > Impala does not support writing to bucketed tables, we need prohibit these > unsupported operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8593) Prohibit write to bucketed table
Yongzhi Chen created IMPALA-8593: Summary: Prohibit write to bucketed table Key: IMPALA-8593 URL: https://issues.apache.org/jira/browse/IMPALA-8593 Project: IMPALA Issue Type: New Feature Reporter: Yongzhi Chen Impala does not support writing to bucketed tables, we need prohibit these unsupported operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8593) Prohibit write to bucketed table
Yongzhi Chen created IMPALA-8593: Summary: Prohibit write to bucketed table Key: IMPALA-8593 URL: https://issues.apache.org/jira/browse/IMPALA-8593 Project: IMPALA Issue Type: New Feature Reporter: Yongzhi Chen Impala does not support writing to bucketed tables, we need prohibit these unsupported operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-7665) Bringing up stopped statestore causes queries to fail
[
https://issues.apache.org/jira/browse/IMPALA-7665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikramjeet Vig resolved IMPALA-7665.
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> Bringing up stopped statestore causes queries to fail
> -
>
> Key: IMPALA-7665
> URL: https://issues.apache.org/jira/browse/IMPALA-7665
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Bikramjeet Vig
>Priority: Critical
> Labels: query-lifecycle, statestore
> Fix For: Impala 3.3.0
>
>
> I can reproduce this by running a long-running query then cycling the
> statestore:
> {noformat}
> tarmstrong@tarmstrong-box:~/Impala/incubator-impala$ impala-shell.sh -q
> "select distinct * from tpch10_parquet.lineitem"
> Starting Impala Shell without Kerberos authentication
> Connected to localhost:21000
> Server version: impalad version 3.1.0-SNAPSHOT DEBUG (build
> c486fb9ea4330e1008fa9b7ceaa60492e43ee120)
> Query: select distinct * from tpch10_parquet.lineitem
> Query submitted at: 2018-10-04 17:06:48 (Coordinator:
> http://tarmstrong-box:25000)
> {noformat}
> If I kill the statestore, the query runs fine, but if I start up the
> statestore again, it fails.
> {noformat}
> # In one terminal, start up the statestore
> $
> /home/tarmstrong/Impala/incubator-impala/be/build/latest/statestore/statestored
> -log_filename=statestored
> -log_dir=/home/tarmstrong/Impala/incubator-impala/logs/cluster -v=1
> -logbufsecs=5 -max_log_files=10
> # The running query then fails
> WARNINGS: Failed due to unreachable impalad(s): tarmstrong-box:22001,
> tarmstrong-box:22002
> {noformat}
> Note that I've seen different subsets impalads reported as failed, e.g.
> "Failed due to unreachable impalad(s): tarmstrong-box:22001"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-7665) Bringing up stopped statestore causes queries to fail
[
https://issues.apache.org/jira/browse/IMPALA-7665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikramjeet Vig resolved IMPALA-7665.
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> Bringing up stopped statestore causes queries to fail
> -
>
> Key: IMPALA-7665
> URL: https://issues.apache.org/jira/browse/IMPALA-7665
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Bikramjeet Vig
>Priority: Critical
> Labels: query-lifecycle, statestore
> Fix For: Impala 3.3.0
>
>
> I can reproduce this by running a long-running query then cycling the
> statestore:
> {noformat}
> tarmstrong@tarmstrong-box:~/Impala/incubator-impala$ impala-shell.sh -q
> "select distinct * from tpch10_parquet.lineitem"
> Starting Impala Shell without Kerberos authentication
> Connected to localhost:21000
> Server version: impalad version 3.1.0-SNAPSHOT DEBUG (build
> c486fb9ea4330e1008fa9b7ceaa60492e43ee120)
> Query: select distinct * from tpch10_parquet.lineitem
> Query submitted at: 2018-10-04 17:06:48 (Coordinator:
> http://tarmstrong-box:25000)
> {noformat}
> If I kill the statestore, the query runs fine, but if I start up the
> statestore again, it fails.
> {noformat}
> # In one terminal, start up the statestore
> $
> /home/tarmstrong/Impala/incubator-impala/be/build/latest/statestore/statestored
> -log_filename=statestored
> -log_dir=/home/tarmstrong/Impala/incubator-impala/logs/cluster -v=1
> -logbufsecs=5 -max_log_files=10
> # The running query then fails
> WARNINGS: Failed due to unreachable impalad(s): tarmstrong-box:22001,
> tarmstrong-box:22002
> {noformat}
> Note that I've seen different subsets impalads reported as failed, e.g.
> "Failed due to unreachable impalad(s): tarmstrong-box:22001"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8592) Add support for insert events for 'LOAD DATA..' statements from Impala.
[ https://issues.apache.org/jira/browse/IMPALA-8592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated IMPALA-8592: - Summary: Add support for insert events for 'LOAD DATA..' statements from Impala. (was: Add support for insert events from 'LOAD DATA..' statements from Impala.) > Add support for insert events for 'LOAD DATA..' statements from Impala. > --- > > Key: IMPALA-8592 > URL: https://issues.apache.org/jira/browse/IMPALA-8592 > Project: IMPALA > Issue Type: Sub-task >Reporter: Anurag Mantripragada >Priority: Major > > Hive generates INSERT events for LOAD DATA.. statements. We should support > the same in Impala. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8592) Add support for insert events from 'LOAD DATA..' statements from Impala.
Anurag Mantripragada created IMPALA-8592: Summary: Add support for insert events from 'LOAD DATA..' statements from Impala. Key: IMPALA-8592 URL: https://issues.apache.org/jira/browse/IMPALA-8592 Project: IMPALA Issue Type: Sub-task Reporter: Anurag Mantripragada Hive generates INSERT events for LOAD DATA.. statements. We should support the same in Impala. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8592) Add support for insert events from 'LOAD DATA..' statements from Impala.
Anurag Mantripragada created IMPALA-8592: Summary: Add support for insert events from 'LOAD DATA..' statements from Impala. Key: IMPALA-8592 URL: https://issues.apache.org/jira/browse/IMPALA-8592 Project: IMPALA Issue Type: Sub-task Reporter: Anurag Mantripragada Hive generates INSERT events for LOAD DATA.. statements. We should support the same in Impala. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (IMPALA-8461) Re-schedule queries if the executor configuration has changed while queued in AC
[ https://issues.apache.org/jira/browse/IMPALA-8461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong reassigned IMPALA-8461: - Assignee: Lars Volker > Re-schedule queries if the executor configuration has changed while queued in > AC > > > Key: IMPALA-8461 > URL: https://issues.apache.org/jira/browse/IMPALA-8461 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.3.0 >Reporter: Lars Volker >Assignee: Lars Volker >Priority: Major > > If the executor configuration changes while a query is waiting to be > admitted, we need to reschedule it. The current behavior tries to run it as > is which will then fail. To achieve this, we should call > Scheduler::Schedule() from the AdmissionController and then re-schedule if > necessary. We need to think about ways to detect changes to the executor > configuration, but a simple hash might be good enough. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5031) UBSAN clean and method for testing UBSAN cleanliness
[ https://issues.apache.org/jira/browse/IMPALA-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850224#comment-16850224 ] ASF subversion and git services commented on IMPALA-5031: - Commit 7ea8c5706db5d2e7424d687465d702f01eef8824 in impala's branch refs/heads/master from Jim Apple [ https://gitbox.apache.org/repos/asf?p=impala.git;h=7ea8c57 ] IMPALA-5031: Out-of-range enum values are undefined behavior This patch handles an undefined behavior in the custom cluster tests in which a reference to an invalid enum value is used. To prevent the invalid value, the bytes are first copied into their underlying integer type. The undefined behavior happens in LdapJdbcTest. The relevant backtrace is: include/c++/4.9.2/bits/stl_algobase.h:199:11: runtime error: load of value 8, which is not a valid value for type 'const TProtocolVersion::type' #0 TProtocolVersion::type const& min(TProtocolVersion::type const&, TProtocolVersion::type const&) include/c++/4.9.2/bits/stl_algobase.h:199:11 #1 ImpalaServer::OpenSession(TOpenSessionResp&, TOpenSessionReq const&) service/impala-hs2-server.cc:304:24 #2 TCLIServiceProcessor::process_OpenSession(int, TProtocol*, TProtocol*, void*) generated-sources/gen-cpp/TCLIService.cpp:4953:13 #3 TCLIServiceProcessor::dispatchCall(TProtocol*, TProtocol*, string const&, int, void*) generated-sources/gen-cpp/TCLIService.cpp:4926:3 #4 ImpalaHiveServer2ServiceProcessor::dispatchCall(TProtocol*, TProtocol*, string const&, int, void*) generated-sources/gen-cpp/ImpalaHiveServer2Service.cpp:505:73 #5 thrift::TDispatchProcessor::process (boost::shared_ptr, boost::shared_ptr, void*) toolchain/thrift-0.9.3-p5/include/thrift/TDispatchProcessor.h:121:12 #6 thrift::server::TAcceptQueueServer::Task::run() rpc/TAcceptQueueServer.cpp:74:26 #7 ThriftThread::RunRunnable(boost::shared_ptr , Promise*) rpc/thrift-thread.cc:74:13 Change-Id: I63379b4c9d2e4738e729a556108c77fed85e6b64 Reviewed-on: http://gerrit.cloudera.org:8080/13438 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > UBSAN clean and method for testing UBSAN cleanliness > > > Key: IMPALA-5031 > URL: https://issues.apache.org/jira/browse/IMPALA-5031 > Project: IMPALA > Issue Type: Task > Components: Backend, Infrastructure >Affects Versions: Impala 2.9.0 >Reporter: Jim Apple >Assignee: Jim Apple >Priority: Minor > > http://releases.llvm.org/3.8.0/tools/clang/docs/UndefinedBehaviorSanitizer.html > builds are supported after https://gerrit.cloudera.org/#/c/6186/, but > Impala's test suite triggers many errors under UBSAN. Those errors should be > fixed and then there should be a way to run the test suite under UBSAN and > fail if there were any errors detected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8383) Bump toolchain version
[ https://issues.apache.org/jira/browse/IMPALA-8383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved IMPALA-8383. - Resolution: Won't Fix > Bump toolchain version > -- > > Key: IMPALA-8383 > URL: https://issues.apache.org/jira/browse/IMPALA-8383 > Project: IMPALA > Issue Type: Bug >Reporter: Hector Acosta >Priority: Major > > The current $IMPALA_TOOLCHAIN_BUILD_ID has a bug where the fastbinary shared > object is missing for some distributions. We should bump the version to an id > that includes fastbinary.so. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[
https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850178#comment-16850178
]
Sahil Takiar commented on IMPALA-8577:
--
Looking around, it seems like
[netty-tcnative|https://netty.io/wiki/forked-tomcat-native.html] is another
popular Java JSSE wrapper around OpenSSL (it uses the SSLContext API - see
[this example|https://netty.io/wiki/forked-tomcat-native.html#wiki-h2-4]). It
seems gRPC and Cassandra both support using netty-tcnative as well:
* [https://docs.datastax.com/en/developer/java-driver/3.3/manual/ssl/#netty]
* [https://github.com/grpc/grpc-java/blob/master/SECURITY.md]
> Crash during OpenSSLSocket.read
> ---
>
> Key: IMPALA-8577
> URL: https://issues.apache.org/jira/browse/IMPALA-8577
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.3.0
>Reporter: David Rorke
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp,
> hs_err_pid6313.log,
> impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313,
>
> impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313
>
>
> Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from
> the stack trace (hs_err log file attached with more complete stack):
> {noformat}
> Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free
> space=8180k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C [impalad+0x2528a33]
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
> unsigned long, int)+0x133
> C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f
> C [impalad+0x266468a] operator delete(void*)+0x32a
> C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d
> J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @
> 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9]
> J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @
> 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c]
> J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @
> 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c]
> J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @
> 0x7f3d4de419cc [0x7f3d4de417c0+0x20c]
> J 2994 C2
> org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult;
> (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134]
> J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @
> 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0]
> J 5090 C2
> com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I
> (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0]
> J 5846 C1
> com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I
> (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384]
> J 5845 C1
> com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31
> bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c]
> {noformat}
> The crash may not be easy to reproduce. I've run this test multiple times
> and only crashed once. I have a core file if needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8577) Crash during OpenSSLSocket.read
[
https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850110#comment-16850110
]
Sahil Takiar commented on IMPALA-8577:
--
[~DanielZhou], [~vishwajeet.dusane] wondering if you have seen any similar
issues when using Wildfly-OpenSSL in ABFS? Specifically, we are seeing crashes
in Impala when enabling Wildfly-OpenSSL in S3A. The crash seems to be due to
memory-corruption (a double free on the same block of memory). I haven't been
able to track down the root cause of the issue, but I'm able to re-produce it
in Impala while running against an
[ASAN|https://en.wikipedia.org/wiki/AddressSanitizer] build.
> Crash during OpenSSLSocket.read
> ---
>
> Key: IMPALA-8577
> URL: https://issues.apache.org/jira/browse/IMPALA-8577
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.3.0
>Reporter: David Rorke
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp,
> hs_err_pid6313.log,
> impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313,
>
> impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313
>
>
> Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from
> the stack trace (hs_err log file attached with more complete stack):
> {noformat}
> Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free
> space=8180k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C [impalad+0x2528a33]
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
> unsigned long, int)+0x133
> C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f
> C [impalad+0x266468a] operator delete(void*)+0x32a
> C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d
> J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @
> 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9]
> J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @
> 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c]
> J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @
> 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c]
> J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @
> 0x7f3d4de419cc [0x7f3d4de417c0+0x20c]
> J 2994 C2
> org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult;
> (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134]
> J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @
> 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0]
> J 5090 C2
> com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I
> (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0]
> J 5846 C1
> com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I
> (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384]
> J 5845 C1
> com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31
> bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c]
> {noformat}
> The crash may not be easy to reproduce. I've run this test multiple times
> and only crashed once. I have a core file if needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8491) Run container as non-root user
[ https://issues.apache.org/jira/browse/IMPALA-8491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8491 started by Tim Armstrong. - > Run container as non-root user > -- > > Key: IMPALA-8491 > URL: https://issues.apache.org/jira/browse/IMPALA-8491 > Project: IMPALA > Issue Type: Sub-task > Components: Infrastructure >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8591) Fuzz test the http endpoint
Thomas Tauber-Marshall created IMPALA-8591: -- Summary: Fuzz test the http endpoint Key: IMPALA-8591 URL: https://issues.apache.org/jira/browse/IMPALA-8591 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.3.0 Reporter: Thomas Tauber-Marshall Assignee: Thomas Tauber-Marshall IMPALA-8538 is adding an http endpoint for clients to connect to and run queries from. The patch for it adds basic functional testing, but we should do additional testing, such as fuzz testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8591) Fuzz test the http endpoint
Thomas Tauber-Marshall created IMPALA-8591: -- Summary: Fuzz test the http endpoint Key: IMPALA-8591 URL: https://issues.apache.org/jira/browse/IMPALA-8591 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.3.0 Reporter: Thomas Tauber-Marshall Assignee: Thomas Tauber-Marshall IMPALA-8538 is adding an http endpoint for clients to connect to and run queries from. The patch for it adds basic functional testing, but we should do additional testing, such as fuzz testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IMPALA-7299) Impala fails to work with the krb5 config 'rdns=false' in Impala 2.12.0/3.0
[ https://issues.apache.org/jira/browse/IMPALA-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849996#comment-16849996 ] ASF subversion and git services commented on IMPALA-7299: - Commit 97ca8d1f4c24c21ba65b036580e27606bf4c939b in impala's branch refs/heads/2.x from Alex Rodoni [ https://gitbox.apache.org/repos/asf?p=impala.git;h=97ca8d1 ] IMPALA-7299: [DOCS] A known issue with IMPALA-7298 Kerberos authentication fails with the reverse DNS lookup disabled. Change-Id: I5b8104a2747b4e8051d4bdcab906486444680218 Reviewed-on: http://gerrit.cloudera.org:8080/10952 Reviewed-by: Sailesh Mukil Tested-by: Impala Public Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/13449 Reviewed-by: Alex Rodoni > Impala fails to work with the krb5 config 'rdns=false' in Impala 2.12.0/3.0 > --- > > Key: IMPALA-7299 > URL: https://issues.apache.org/jira/browse/IMPALA-7299 > Project: IMPALA > Issue Type: Task > Components: Docs >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Sailesh Mukil >Assignee: Alex Rodoni >Priority: Critical > Labels: docs > Fix For: Impala 3.1.0 > > > Since we've switched to using KRPC from Impala 2.12.0 and Impala 3.0, we > found a bug that doesn't allow for kerberized communication if the following > flag in krb5.conf is changed from its default of 'rdns=true' to 'rdns=false'. > The current workaround is to set it back to its default of 'true'. Keep in > mind that the 'dns_canonicalize_hostname' flag should also be 'true' > otherwise the 'rdns' flag will be ignored. > More details on the flag can be found here: > http://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.html > A fix for this issue is tracked by IMPALA-7298. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8438) List valid writeIds for a ACID table
[ https://issues.apache.org/jira/browse/IMPALA-8438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1685#comment-1685 ] ASF subversion and git services commented on IMPALA-8438: - Commit 3bbb85598a271e18e512b066ff22c263f85ab241 in impala's branch refs/heads/master from Zoltan Borok-Nagy [ https://gitbox.apache.org/repos/asf?p=impala.git;h=3bbb855 ] IMPALA-8585: Fix StmtMetadataLoaderTest when compiled against Hive 3 IMPALA-8438 added a new event to the metadata loading timeline when Impala is integrated with Hive 3. This updates an assertion in a test about the number of events during metadata loading. Change-Id: I0028c22a122a31ab08d863e90658271f1274e9a4 Reviewed-on: http://gerrit.cloudera.org:8080/13444 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > List valid writeIds for a ACID table > > > Key: IMPALA-8438 > URL: https://issues.apache.org/jira/browse/IMPALA-8438 > Project: IMPALA > Issue Type: New Feature > Components: Frontend >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Critical > Labels: impala-acid > Fix For: Impala 3.3.0 > > > Before listing the partitions of a table, fetch and store the list of valid > (committed) writeIds for the table. This will be used later during > planning/refresh. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7298) Don't pass resolved IP address as hostname when creating proxy
[
https://issues.apache.org/jira/browse/IMPALA-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849997#comment-16849997
]
ASF subversion and git services commented on IMPALA-7298:
-
Commit 97ca8d1f4c24c21ba65b036580e27606bf4c939b in impala's branch
refs/heads/2.x from Alex Rodoni
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=97ca8d1 ]
IMPALA-7299: [DOCS] A known issue with IMPALA-7298
Kerberos authentication fails with the reverse DNS lookup disabled.
Change-Id: I5b8104a2747b4e8051d4bdcab906486444680218
Reviewed-on: http://gerrit.cloudera.org:8080/10952
Reviewed-by: Sailesh Mukil
Tested-by: Impala Public Jenkins
Reviewed-on: http://gerrit.cloudera.org:8080/13449
Reviewed-by: Alex Rodoni
> Don't pass resolved IP address as hostname when creating proxy
> --
>
> Key: IMPALA-7298
> URL: https://issues.apache.org/jira/browse/IMPALA-7298
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 2.12.0, Impala 3.1.0
>Reporter: Michael Ho
>Assignee: Michael Ho
>Priority: Critical
> Fix For: Impala 3.1.0
>
>
> {{KrpcDataStreamSender}} passes a resolved IP address when creating a proxy.
> Instead, we should pass both the resolved address and the hostname when
> creating the proxy so that we won't end up using the IP address as the
> hostname in the Kerberos principal.
> Due to the oversight above, the following error may show up when running a
> build of 2.12.0 when a user has Kerberos enabled and specified
> {{impala/@}} as the kerberos principal.
> {noformat}
> WARNINGS: TransmitData() to X.X.X.X:27000 failed: Not authorized: Client
> connection negotiation failed: client connection to X.X.X.X:27000: Server
> impala/x.x@vpc.cloudera.com not found in Kerberos database
> {noformat}
> The workaround for this problem is to have {{rdns=true}} in
> {{/etc/krb5.conf}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8588) FIx revoke grant option behavior
[ https://issues.apache.org/jira/browse/IMPALA-8588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8588 started by Austin Nobis. > FIx revoke grant option behavior > > > Key: IMPALA-8588 > URL: https://issues.apache.org/jira/browse/IMPALA-8588 > Project: IMPALA > Issue Type: Sub-task > Components: Catalog, Frontend >Reporter: Austin Nobis >Assignee: Austin Nobis >Priority: Critical > Fix For: Impala 3.3.0 > > > Given permissions have been granted via: > *grant select on database functional for user admin with grant option;* > > > Currently, running the following statement will remove the *grant option*: > *revoke select on database functional from user admin;* > > The following statement will also not remove the *grant option:* > *revoke grant option for select on database functional from user admin;* > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6903) Allow download of text profile via Impala WebUI
[ https://issues.apache.org/jira/browse/IMPALA-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849577#comment-16849577 ] ASF subversion and git services commented on IMPALA-6903: - Commit b00d031fb37b83924d4dfa020360f35540717501 in impala's branch refs/heads/master from Yongzhi Chen [ https://gitbox.apache.org/repos/asf?p=impala.git;h=b00d031 ] IMPALA-6903: Download profile from WebUI in text format Add a link called "Download Text Profile" to profile tab. The link allows users to download runtime profiles as UTF-8 encoded file. Get text profile from backend by making http request. Tests: Manually tested to check downloaded files. Ran all core tests. Add test_download_text_profile to test_web_pages.py. Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b Reviewed-on: http://gerrit.cloudera.org:8080/1 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Allow download of text profile via Impala WebUI > --- > > Key: IMPALA-6903 > URL: https://issues.apache.org/jira/browse/IMPALA-6903 > Project: IMPALA > Issue Type: Improvement >Affects Versions: Impala 2.11.0 >Reporter: Gabor Kaszab >Assignee: Yongzhi Chen >Priority: Minor > Labels: newbie > Attachments: download-text-profile-link-screenshot.png > > > In Impala WebUI it's already possible to download the query profile in thrift > format (https://issues.apache.org/jira/browse/IMPALA-2555). It would be nice > to have the same download option for text format as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8524) Avoid calling "hive" via command line in EE tests
[ https://issues.apache.org/jira/browse/IMPALA-8524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849575#comment-16849575 ] ASF subversion and git services commented on IMPALA-8524: - Commit 6839d9738b0d5c3c176fea4345ab623783915454 in impala's branch refs/heads/master from Zoltan Borok-Nagy [ https://gitbox.apache.org/repos/asf?p=impala.git;h=6839d97 ] IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests "hive -e SQL..." without further parameters no longer works when USE_CDP_HIVE=true (it doesn't establish a connection). Some tests used this to load data. part2: there were some places that still called hive without a beeline connection. I had to break up views-compatibility.test for the different versions of Hive. Change-Id: Ia45b64cc1da78190e6f239a5f462308d7fa56f4b Reviewed-on: http://gerrit.cloudera.org:8080/13402 Reviewed-by: Zoltan Borok-Nagy Tested-by: Impala Public Jenkins > Avoid calling "hive" via command line in EE tests > - > > Key: IMPALA-8524 > URL: https://issues.apache.org/jira/browse/IMPALA-8524 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Reporter: Csaba Ringhofer >Priority: Major > Fix For: Impala 3.3.0 > > > "hive -e SQL..." without further parameters no longer works when > USE_CDP_HIVE=true (it doesn't establish a connection). Some tests used this > to load data. > These calls can be replaced with ImpalaTestSuite.run_stmt_in_hive() which > seems like a good idea regardless of the Hive 3 efforts. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8400) Implement Ranger audit event handler
[ https://issues.apache.org/jira/browse/IMPALA-8400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849574#comment-16849574 ] ASF subversion and git services commented on IMPALA-8400: - Commit 2750f0ab358ce754fb651a019fc06a071395395f in impala's branch refs/heads/master from Fredy Wijaya [ https://gitbox.apache.org/repos/asf?p=impala.git;h=2750f0a ] IMPALA-8400: Implement Ranger audit event handler This patch implements Ranger audit event handler to behave similarly to the Hive/Ranger audit event handler, most notably: - Buffer the audit events during authorization and only flush them once the authorization is complete. - The audit will only add the event for the first failure. - Create an audit event handler per statement. Testing: - Added test cases in RangerAuditLogTest - Ran FE tests - Ran all E2E authorization tests Change-Id: Ife93c3e708875ef5fc0117153ad8ee225a88518b Reviewed-on: http://gerrit.cloudera.org:8080/13309 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Implement Ranger audit event handler > > > Key: IMPALA-8400 > URL: https://issues.apache.org/jira/browse/IMPALA-8400 > Project: IMPALA > Issue Type: Sub-task > Components: Catalog, Frontend >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Critical > Fix For: Impala 3.3.0 > > > The current implementation logs the audit log per request. We should consider > doing the audit log atomically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8248) Improve Ranger test coverage
[ https://issues.apache.org/jira/browse/IMPALA-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849576#comment-16849576 ] ASF subversion and git services commented on IMPALA-8248: - Commit 377471fb4b2794caae63601f937b6c4a48ef2cd3 in impala's branch refs/heads/master from Austin Nobis [ https://gitbox.apache.org/repos/asf?p=impala.git;h=377471f ] IMPALA-8248: Improve Ranger test coverage This patch adds increased coverage for Apache Ranger integration. Specifically, tests were added that interact directly with Apache Ranger via the REST API and then assertions were made against Impala to test proper behavior. Testing: - Ran all E2E authorization tests - Added a test that adds hive privileges to Ranger and verifies they do not show in Impala. - Added a test that grants privileges to Ranger via the UI and runs "refresh authorization" to verify they exist in Impala. Change-Id: I15ce57ea96fbf6bff9bcabf7300fbadea7c55b09 Reviewed-on: http://gerrit.cloudera.org:8080/13413 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Improve Ranger test coverage > > > Key: IMPALA-8248 > URL: https://issues.apache.org/jira/browse/IMPALA-8248 > Project: IMPALA > Issue Type: Sub-task > Components: Infrastructure >Reporter: Fredy Wijaya >Assignee: Austin Nobis >Priority: Critical > Fix For: Impala 3.3.0 > > > We have authorization tests that are specific to Sentry and authorization > tests that can be applicable to any authorization provider. We need to > re-organize the authorization tests to easily differentiate between > Sentry-specific tests vs generic authorization tests. > This also will improve test coverage for Ranger. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8572) Move query hook execution to before query unregistration
[ https://issues.apache.org/jira/browse/IMPALA-8572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849579#comment-16849579 ] ASF subversion and git services commented on IMPALA-8572: - Commit 31195eb8119ac6a557486a10dc24692bb0202f85 in impala's branch refs/heads/master from Radford Nguyen [ https://gitbox.apache.org/repos/asf?p=impala.git;h=31195eb ] IMPALA-8473: Publish lineage info via hook This commit introduces a hook mechanism for publishing, lineage data specifically, but query information more generally, from Impala. The legacy behavior of writing the lineage file is being retained but deprecated. Hooks can be implemented by downstream consumers (i.e. runtime dependencies) to hook into supported places during Impala query execution: - impalad startup - query completion - see IMPALA-8572 for caveat/details The consumers are to be frontend Java dependencies intiated at runtime. 2 backend flags configure this behavior: - `query_event_hook_classes` specifies a comma-separated list of hook consumer implementation classes that are instantiated and registered at impala start up. - `query_event_hook_nthreads` specifies the number of threads to use for asynchronous hook execution. (Relevant if multiple hooks are registered.) Lineage information is passed from the backend after a query completes (but before it returns) and given to every hook to execute asynchronously. In other words, a query may complete and return to the user before any or all hooks have completed executing. An exception during hook on-query-complete execution will simply be logged and will not be (directly) fatal to the system. Tests: - added unit tests for FE hook execution - added E2E tests for hook configuration, execution, error - ran full build, tests Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 Reviewed-on: http://gerrit.cloudera.org:8080/13352 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Move query hook execution to before query unregistration > > > Key: IMPALA-8572 > URL: https://issues.apache.org/jira/browse/IMPALA-8572 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: radford nguyen >Priority: Major > > Placeholder: description coming soon -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8585) Impala ACID tests
[ https://issues.apache.org/jira/browse/IMPALA-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849580#comment-16849580 ] ASF subversion and git services commented on IMPALA-8585: - Commit 5faf1745b0e27ff3f95d532ad5e7dc8e34310172 in impala's branch refs/heads/master from Csaba Ringhofer [ https://gitbox.apache.org/repos/asf?p=impala.git;h=5faf174 ] IMPALA-8585: Fix for upgraded + compacted acid tables Tables that already had data before altered to be an ACID table keep the old data in their root table/partition directory if hive.mm.allow.originals == true. These files should be merged to the base file during the first compaction, so should be read only if there is no valid base yet. Also added EE tests for upgraded tables. Change-Id: I062d8e76f90e0da1b954bf156208c0afb424deb1 Reviewed-on: http://gerrit.cloudera.org:8080/13427 Reviewed-by: Zoltan Borok-Nagy Tested-by: Impala Public Jenkins > Impala ACID tests > - > > Key: IMPALA-8585 > URL: https://issues.apache.org/jira/browse/IMPALA-8585 > Project: IMPALA > Issue Type: Improvement >Reporter: Zoltán Borók-Nagy >Priority: Critical > Labels: impala-acid > > Umbrella Jira for adding tests about ACID functionality, e.g.: > * Ordinary table that was upgraded to ACID table > * Inserting data in hive and querying it in Impala concurrently > * Compute stats interoperability between Hive and Impala > * Partitioned tables, dynamic partitioning -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8473) Refactor lineage publication mechanism to allow for different consumers
[
https://issues.apache.org/jira/browse/IMPALA-8473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849578#comment-16849578
]
ASF subversion and git services commented on IMPALA-8473:
-
Commit 31195eb8119ac6a557486a10dc24692bb0202f85 in impala's branch
refs/heads/master from Radford Nguyen
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=31195eb ]
IMPALA-8473: Publish lineage info via hook
This commit introduces a hook mechanism for publishing,
lineage data specifically, but query information more
generally, from Impala.
The legacy behavior of writing the lineage file is
being retained but deprecated.
Hooks can be implemented by downstream consumers (i.e.
runtime dependencies) to hook into supported places during
Impala query execution:
- impalad startup
- query completion
- see IMPALA-8572 for caveat/details
The consumers are to be frontend Java dependencies
intiated at runtime. 2 backend flags configure this
behavior:
- `query_event_hook_classes` specifies a comma-separated
list of hook consumer implementation classes that
are instantiated and registered at impala start up.
- `query_event_hook_nthreads`
specifies the number of threads to use for asynchronous
hook execution. (Relevant if multiple hooks are
registered.)
Lineage information is passed from the backend after
a query completes (but before it returns) and given
to every hook to execute asynchronously. In other words,
a query may complete and return to the user before any
or all hooks have completed executing. An exception
during hook on-query-complete execution will simply be logged
and will not be (directly) fatal to the system.
Tests:
- added unit tests for FE hook execution
- added E2E tests for hook configuration, execution, error
- ran full build, tests
Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1
Reviewed-on: http://gerrit.cloudera.org:8080/13352
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Refactor lineage publication mechanism to allow for different consumers
> ---
>
> Key: IMPALA-8473
> URL: https://issues.apache.org/jira/browse/IMPALA-8473
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend, Frontend
>Reporter: radford nguyen
>Assignee: radford nguyen
>Priority: Critical
> Attachments: ImpalaPostExecHook-infra.patch
>
>
> Impetus for this change is to allow lineage to be consumed by Atlas via Kafka.
> h3. Design Proposal
> Implement a plugin approach (similar to {{authorization_provider}}) for
> consuming query event hooks, where downstream users can provide their own
> hook implementations as runtime dependencies.
> Keep but deprecate existing lineage event file writing.
> [~mad...@apache.org] has provided a fe patch (attached) with suggested
> mechanism for allowing multiple hooks to be registered with the fe. Hooks
> would be invoked from the be at appropriate places, e.g.
> [https://github.com/apache/impala/blob/c1b0a073938c144e9bf33901bd4df6dcda0f09ec/be/src/service/impala-server.cc#L466].
> The hooks should all be executed asynchronously, so the current thinking is
> that this execution should happen in the fe, since the be does not know about
> what hooks are registered. IOW, the
> {{ImpalaPostExecHookFactory.executeHooks}} method (see patch) should probably
> make use of a thread-pool executor service (or something similar) in order to
> execute all hooks in parallel and in a non-blocking manner, returning to the
> be asap.
>
> h3. Code Review
> [https://gerrit.cloudera.org/#/c/13352/]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8369) Impala should be able to interoperate with Hive 3.1.0
[ https://issues.apache.org/jira/browse/IMPALA-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849581#comment-16849581 ] ASF subversion and git services commented on IMPALA-8369: - Commit 5ce57cafb20f4627054c6c1d0d2a79e9ae97fd78 in impala's branch refs/heads/master from Csaba Ringhofer [ https://gitbox.apache.org/repos/asf?p=impala.git;h=5ce57ca ] IMPALA-8369: Add HIVE_MAJOR_VERSION section to planner tests + some fixes Hive 3 creates different number of files for some tables than Hive2, which broke some test cases in resource-requirements.test. The fix is to run different versions of these tests depending on Hive version. This is done by adding a new section HIVE_MAJOR_VERSION, which leads to skipping the given test case if the Hive version is different in the cluster, e.g.: --- HIVE_MAJOR_VERSION 3 Change-Id: Ied7ba7911da23cbca12149e062f4e1a444613a36 Reviewed-on: http://gerrit.cloudera.org:8080/13414 Tested-by: Impala Public Jenkins Reviewed-by: Csaba Ringhofer > Impala should be able to interoperate with Hive 3.1.0 > - > > Key: IMPALA-8369 > URL: https://issues.apache.org/jira/browse/IMPALA-8369 > Project: IMPALA > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: impala-acid > > Currently, Impala only works with Hive 2.1.1. Since Hive 3.1.0 has been > released for a while it would be good to add support for Hive 3.1.0 (HMS > 3.1.0). This patch will focus on ability to connect to HMS 3.1.0 and run > existing tests. It will not focus on adding support for newer features like > ACID in Hive 3.1.0 which can be taken up as separate JIRA. > It would be good to make changes to Impala source code such that it can work > with both Hive 2.1.0 and Hive 3.1.0 without the need to create a separate > branch. However, this should be a aspirational goal. If we hit a blocker we > should investigate alternative approaches. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7295) Remove IMPALA_MINICLUSTER_PROFILE=2
[ https://issues.apache.org/jira/browse/IMPALA-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849573#comment-16849573 ] ASF subversion and git services commented on IMPALA-7295: - Commit 33d58e42efab7a2919a710c2916bc92ef826a14e in impala's branch refs/heads/2.x from stiga-huang [ https://gitbox.apache.org/repos/asf?p=impala.git;h=33d58e4 ] Ignore "IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2" Change-Id: I0eb5afd1db51cdc6b4a3dd6091d2926a72400e1a Reviewed-on: http://gerrit.cloudera.org:8080/13371 Reviewed-by: Fredy Wijaya Tested-by: Quanlong Huang > Remove IMPALA_MINICLUSTER_PROFILE=2 > --- > > Key: IMPALA-7295 > URL: https://issues.apache.org/jira/browse/IMPALA-7295 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.0 >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Major > Fix For: Impala 3.1.0 > > > Based on the discussion here: > https://lists.apache.org/thread.html/49f9b68ed3d6d2c0fdee16a877b259922545e4824e1233479227a657@%3Cdev.impala.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
