[jira] [Resolved] (IMPALA-6576) Add metric for Data Stream Service Queue memory consumption
[ https://issues.apache.org/jira/browse/IMPALA-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Volker resolved IMPALA-6576. - Resolution: Fixed Fix Version/s: Impala 2.12.0 Impala 3.0 > Add metric for Data Stream Service Queue memory consumption > --- > > Key: IMPALA-6576 > URL: https://issues.apache.org/jira/browse/IMPALA-6576 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Lars Volker >Assignee: Lars Volker >Priority: Major > Labels: rpc > Fix For: Impala 3.0, Impala 2.12.0 > > > We should add a metric exposing the memory usage of the Data Stream Service > Queue memory tracker. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-6651) Impala Doc: Fine-grained privileges
Alex Rodoni created IMPALA-6651: --- Summary: Impala Doc: Fine-grained privileges Key: IMPALA-6651 URL: https://issues.apache.org/jira/browse/IMPALA-6651 Project: IMPALA Issue Type: Sub-task Components: Docs Reporter: Alex Rodoni Assignee: Alex Rodoni -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-6633) Warn during startup when running on kernel version that triggers IMPALA-6351
[ https://issues.apache.org/jira/browse/IMPALA-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Volker resolved IMPALA-6633. - Resolution: Won't Fix Fix Version/s: Not Applicable After more consideration I think this should be caught by whatever management solution one uses to manage their servers, as it will likely also affect other software on the system. It seems suboptimal to rely on Impala to warn about this. > Warn during startup when running on kernel version that triggers IMPALA-6351 > > > Key: IMPALA-6633 > URL: https://issues.apache.org/jira/browse/IMPALA-6633 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.12.0 >Reporter: Lars Volker >Priority: Major > Labels: ramp-up > Fix For: Not Applicable > > > We should detect kernel versions that trigger IMPALA-6351 and issue a warning > during startup. We could even consider aborting with an error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-6609) Some COUNTER_ADD() in KrpcDataStreamRecvr may lead to use-after-free
[
https://issues.apache.org/jira/browse/IMPALA-6609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-6609.
Resolution: Fixed
Fix Version/s: Impala 2.12.0
Impala 3.0
> Some COUNTER_ADD() in KrpcDataStreamRecvr may lead to use-after-free
>
>
> Key: IMPALA-6609
> URL: https://issues.apache.org/jira/browse/IMPALA-6609
> Project: IMPALA
> Issue Type: Sub-task
> Components: Distributed Exec
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Michael Ho
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.0, Impala 2.12.0
>
>
> The following line in {{KrpcDataStreamRecvr}} may lead to use-after-free. A
> {{KrpcDataStreamRecvr}} is co-owned by {{KrpcDataStreamMgr}} and an
> {{ExchangeNode}}. There is a window in which the {{ExchangeNode}} has already
> been closed and when the last reference to {{KrpcDataStreamMgr}} goes away.
> In this window, the {{KrpcDataStreamRecvr}} queues should all have been
> cancelled and closed but the receiver itself may still reference resources
> owned by the {{ExchangeNode}}. The general pattern is that once a receiver's
> queue is cancelled or closed, there should be no more access to its data
> structures not owned by the receiver or the queue itself. However, there are
> a couple of places in {{KrpcDataStreamRecvr}} which violate this pattern:
> {noformat}
> void KrpcDataStreamRecvr::RemoveSender(int sender_id) {
> int use_sender_id = is_merging_ ? sender_id : 0;
> sender_queues_[use_sender_id]->DecrementSenders();
> COUNTER_ADD(num_eos_received_, 1); <<---
> }
> {noformat}
> In the long run, we really need to re-think this business of co-ownership and
> simplify the lifecycle management of a {{KrpcDataStreamRecvr}} object.
> The bug above may lead to crash like the following:
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x018639e9, pid=37254, tid=0x7f490222b700
> #
> # JRE version: OpenJDK Runtime Environment (8.0_151-b12) (build
> 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
> # Java VM: OpenJDK 64-Bit Server VM (25.151-b12 mixed mode linux-amd64
> compressed oops)
> # Problematic frame:
> # C [impalad+0x14639e9] impala::KrpcDataStreamRecvr::RemoveSender(int)+0x73
> #
> # Core dump written. Default location: /home/ubuntu/Impala/core or core.37254
> #
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-6650) Add fine-grained DROP privilege
Fredy Wijaya created IMPALA-6650: Summary: Add fine-grained DROP privilege Key: IMPALA-6650 URL: https://issues.apache.org/jira/browse/IMPALA-6650 Project: IMPALA Issue Type: Improvement Reporter: Fredy Wijaya Currently ALL privilege is required to drop a database/table. We need to have a more fine-grained privilege, i.e. DROP to drop a database/table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-6649) Add fine-grained ALTER privilege
Fredy Wijaya created IMPALA-6649: Summary: Add fine-grained ALTER privilege Key: IMPALA-6649 URL: https://issues.apache.org/jira/browse/IMPALA-6649 Project: IMPALA Issue Type: Improvement Reporter: Fredy Wijaya Currently ALL privilege is required to alter a table. We need to have a more fine-grained privilege, i.e. ALTER to alter a table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-6648) Add fine-grained privileges to role
Fredy Wijaya created IMPALA-6648: Summary: Add fine-grained privileges to role Key: IMPALA-6648 URL: https://issues.apache.org/jira/browse/IMPALA-6648 Project: IMPALA Issue Type: Epic Reporter: Fredy Wijaya Add CREATE, ALTER, DROP, and REFRESH fine-grained privileges. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-6647) Add fine-grained privileges to role
Fredy Wijaya created IMPALA-6647: Summary: Add fine-grained privileges to role Key: IMPALA-6647 URL: https://issues.apache.org/jira/browse/IMPALA-6647 Project: IMPALA Issue Type: Improvement Components: Frontend Reporter: Fredy Wijaya Add CREATE, ALTER, DROP, and REFRESH fine-grained privileges to role. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-6646) Add a way to test with MySQL backed metastore
Zach Amsden created IMPALA-6646: --- Summary: Add a way to test with MySQL backed metastore Key: IMPALA-6646 URL: https://issues.apache.org/jira/browse/IMPALA-6646 Project: IMPALA Issue Type: New Feature Reporter: Zach Amsden Impala's minicluster doesn't have a way to start up with the Hive metastore running on MySQL. As we've found out, (see IMPALA-6389) there are a couple of differences in behavior between Postgresql and MySQL backed HMS, and it would be good to have a way to reproduce and test these cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-6637) SIGSEGV at impala::BufferedTupleStream::FixUpStringsForRead()
[
https://issues.apache.org/jira/browse/IMPALA-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-6637.
Resolution: Not A Bug
Fix Version/s: Not Applicable
Based on the analysis so far, there is a particular node in the cluster which
always produces incorrect number of rows when executing a fragment with a PHJ
node with one or more partitions. The kernel log on that node suggests there is
sign of disk failure. Will retry the experiment after that node is fixed.
Closing for now.
> SIGSEGV at impala::BufferedTupleStream::FixUpStringsForRead()
> --
>
> Key: IMPALA-6637
> URL: https://issues.apache.org/jira/browse/IMPALA-6637
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Tim Armstrong
>Priority: Blocker
> Labels: crash
> Fix For: Not Applicable
>
>
> When running stress test in a 140 node cluster, one of the node crashed with
> in {{BufferedTupleStream::FixUpStringsForRead()}}
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x01212607, pid=22971, tid=140176345585408
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build
> 1.7.0_67-b01)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # C [impalad+0xe12607]
> impala::BufferedTupleStream::FixUpStringsForRead(std::vector std::allocator > const&, impala::Tuple*)+0x17
> #
> # Core dump written. Default location: /var/log/impalad/core or core.22971
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.sun.com/bugreport/crash.jsp
> #
> --- T H R E A D ---
> Current thread is native thread
> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
> si_addr=0xc4cd450a
> Registers:
> RAX=0x7fbe4d9f8298, RBX=0x7fc4ca272750, RCX=0x7fc5f914a630,
> RDX=0xc4cd44da
> RSP=0x7f7d5948c3a0, RBP=0x7f7d5948c3a0, RSI=0x0030,
> RDI=0x7fc03b299b00
> R8 =0xc4cd451c, R9 =0x7fbe4d9f82a0, R10=0x0001,
> R11=0x0001
> R12=0x0075, R13=0x7fc4ca272760, R14=0x0001,
> R15=0x7fc03b299b00
> RIP=0x01212607, EFLAGS=0x00010283, CSGSFS=0x27c40033,
> ERR=0x0005
> TRAPNO=0x000e
> Top of Stack: (sp=0x7f7d5948c3a0)
> 0x7f7d5948c3a0: 7f7d5948c420 01213071
> 0x7f7d5948c3b0: 7f7d5948c46f 7fc4a1753d40
> 0x7f7d5948c3c0: 7f7d5948c470 7fc03b299bb8
> 0x7f7d5948c3d0: 0010 0400a1753d40
> 0x7f7d5948c3e0: 7fc4a1753d40 7fbd8b945ac0
> 0x7f7d5948c3f0: 7f7d5948c540 7fc790376000
> 0x7f7d5948c400: 7fbd8b945ac0 7f7d5948c540
> 0x7f7d5948c410: 7fc92fde 7fc790376358
> 0x7f7d5948c420: 7f7d5948c4b0 01042351
> 0x7f7d5948c430: 0001 7fbd8b945ac0
> 0x7f7d5948c440: 7f7d5948c540 7fc92fde
> 0x7f7d5948c450: 7f7d5948c460 7fcf0b06aa47
> 0x7f7d5948c460: 7f7d5948c490 0035d9a03ea6
> 0x7f7d5948c470: 7f7d5948c540 7f7d5948c4b0
> 0x7f7d5948c480: 7f7d5948c4b0 7fc790376000
> 0x7f7d5948c490: 7fbd8b945ac0 7f7d5948c540
> 0x7f7d5948c4a0: 7fc92fde 7fc790376358
> 0x7f7d5948c4b0: 7f7d5948c5a0 01047e13
> 0x7f7d5948c4c0: 7fc92fde 0001
> 0x7f7d5948c4d0: 7f7d5948c500 7f7d5948c4f0
> 0x7f7d5948c4e0: 7f7d5948c5de 7f7d5948c5f0
> 0x7f7d5948c4f0: 0400
> 0x7f7d5948c500: 02470482 231d2d90
> 0x7f7d5948c510: 0087be8ed8dfc0af
> 0x7f7d5948c520: 7f7d5948c601
> 0x7f7d5948c530: 7fc6bbaf2ea8
> 0x7f7d5948c540: 0087be8ed8dfc190 016e
> 0x7f7d5948c550: 7f7d5948c600
> 0x7f7d5948c560: 7fb7e8b7b5c0
> 0x7f7d5948c570: 7f7d5948c580 1d916200
> 0x7f7d5948c580: 7f7d5948c600 7fb4efbf2f40
> 0x7f7d5948c590: 7f7d5948c5f0 7f7d5948c5de
> Instructions: (pc=0x01212607)
> 0x012125e7: 2e 0f 1f 84 00 00 00 00 00 55 4c 8b 4e 08 48 8b
> 0x012125f7: 06 48 89 e5 4c 39 c8 74 3a 48 8b 08 48 63 71 74
> 0x01212607: 44 0f be 04 32 0f b6 71 78 41 85 f0 75 1c 48 63
> 0x01212617: 49 70 48 8b b7 b8 00 00 00 48 01 d1 48 89 31 48
>
[jira] [Created] (IMPALA-6645) Consider enabling disk spill encryption by default
Tim Armstrong created IMPALA-6645: - Summary: Consider enabling disk spill encryption by default Key: IMPALA-6645 URL: https://issues.apache.org/jira/browse/IMPALA-6645 Project: IMPALA Issue Type: Improvement Components: Backend Affects Versions: Not Applicable Reporter: Tim Armstrong Assignee: Tim Armstrong We currently have integrity checks disabled by default because of the additional CPU cost. Now that IMPALA-6219, the integrity checks are cheaper so it may be worth just paying the cost. We could also consider doing a lighter-weight CRC checksum, but AES-GCM may be fast enough at 2.3GB/s to do by default. We should do an experiment to see if there's a major impact on spill-to-disk perf. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-6639) Crash with 'ORDER BY' in 'OVER' clause with 'RANDOM'
[
https://issues.apache.org/jira/browse/IMPALA-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Behm resolved IMPALA-6639.
Resolution: Duplicate
> Crash with 'ORDER BY' in 'OVER' clause with 'RANDOM'
>
>
> Key: IMPALA-6639
> URL: https://issues.apache.org/jira/browse/IMPALA-6639
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Balazs Jeszenszky
>Priority: Blocker
> Labels: crash
>
> The following query crashes Impala reliably:
> {code:java}
> select AVG(n) OVER(ORDER By n) from (
> select RANDOM() as n from (select 1 union all select 1) a) b;
> {code}
> Stack trace:
> {code:java}
> #0 0x7f98565315e5 in raise () from /lib64/libc.so.6
> #1 0x7f9856532dc5 in abort () from /lib64/libc.so.6
> #2 0x7f9858697a55 in os::abort(bool) () from
> /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
> #3 0x7f9858817f87 in VMError::report_and_die() ()
>from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
> #4 0x7f985869c96f in JVM_handle_linux_signal ()
>from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
> #5
> #6 0x02c9ed73 in impala::Sorter::Run::Run (this=0x94612a0,
> parent=0xa396080, sort_tuple_desc=0x8d9a750,
> initial_run=true) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/sorter.cc:624
> #7 0x02ca5c00 in impala::Sorter::Open (this=0xa396080)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/sorter.cc:1551
> #8 0x02901a09 in impala::SortNode::Open (this=0x945de00,
> state=0x9d92180)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/sort-node.cc:82
> #9 0x02919bcb in impala::AnalyticEvalNode::Open (this=0x6dbb100,
> state=0x9d92180)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/analytic-eval-node.cc:187
> #10 0x01893d03 in impala::FragmentInstanceState::Open (this=0xae64760)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:255
> #11 0x018917fd in impala::FragmentInstanceState::Exec (this=0xae64760)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:80
> #12 0x0187a7ac in impala::QueryState::ExecFInstance (this=0x8fd0d00,
> fis=0xae64760)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/query-state.cc:382
> #13 0x0187906e in impala::QueryState::::operator()(void)
> const (__closure=0x7f97ffd88bc8)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/query-state.cc:325
> #14 0x0187b3eb in
> boost::detail::function::void_function_obj_invoker0,
> void>::invoke(boost::detail::function::function_buffer &)
> (function_obj_ptr=...)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #15 0x017c88de in boost::function0::operator()
> (this=0x7f97ffd88bc0)
> at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #16 0x01abe113 in impala::Thread::SuperviseThread (name=...,
> category=..., functor=...,
> thread_started=0x7f9800788ab0) at
> /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/util/thread.cc:352
> #17 0x01ac6c9e in
> boost::_bi::list4 std::char_traits, std::allocator > >,
> boost::_bi::value std::allocator > >, boost::_bi::value >,
> boost::_bi::value >::operator() std::basic_string&, const std::basic_string&,
> boost::function, impala::Promise*),
> boost::_bi::list0>(boost::_bi::type, void (*&)(const
> std::basic_string &,
> const std::basic_string
> &, boost::function, impala::Promise *), boost::_bi::list0 &,
> int) (this=0xa3901c0, f=@0xa3901b8, a=...)
> {code}
> sort_tuple_size_ ends up as zero in this division:
> https://github.com/cloudera/Impala/blob/cdh5-2.11.0_5.14.0/be/src/runtime/sorter.cc#L624
> Looks like the tuple descriptor is malformed:
> {code:java}
> (gdb) p *(impala::TupleDescriptor *) sort_tuple_desc
> $4 = {static LLVM_CLASS_NAME = 0x3d649c4 "class.impala::TupleDescriptor", id_
> = 8, table_desc_ = 0x0,
> byte_size_ = 0, num_null_bytes_ = 0, null_bytes_offset_ = 0,
> slots_ = { std::allocator >> = {
> _M_impl = {> =
> {<__gnu_cxx::new_allocator> = {},
>
[jira] [Resolved] (IMPALA-6629) Clearer and more concise logging during catalog topic updates
[
https://issues.apache.org/jira/browse/IMPALA-6629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianyi Wang resolved IMPALA-6629.
-
Resolution: Fixed
Fix Version/s: Impala 2.13.0
Impala 3.0
> Clearer and more concise logging during catalog topic updates
> -
>
> Key: IMPALA-6629
> URL: https://issues.apache.org/jira/browse/IMPALA-6629
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
>Affects Versions: Impala 2.12.0
>Reporter: Alexander Behm
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: supportability
> Fix For: Impala 3.0, Impala 2.13.0
>
>
> We should improve the logging on the catalogd side during a catalog topic
> update.
> Snippet of current log:
> {code}
> I0307 21:23:41.011883 21342 catalog-server.cc:477] Publishing update:
> TABLE:functional_avro_def.nulltable original size: 65
> I0307 21:23:41.011922 21342 CatalogServiceCatalog.java:426] Collected catalog
> update: TABLE:functional_avro_def.alltypestiny version: 134
> I0307 21:23:41.011927 21342 catalog-server.cc:477] Publishing update:
> TABLE:functional_avro_def.alltypestiny original size: 68
> I0307 21:23:41.011975 21342 CatalogServiceCatalog.java:426] Collected catalog
> update: TABLE:functional_avro_def.jointbl version: 138
> I0307 21:23:41.011986 21342 catalog-server.cc:477] Publishing update:
> TABLE:functional_avro_def.jointbl original size: 63
> I0307 21:23:41.012027 21342 CatalogServiceCatalog.java:426] Collected catalog
> update: TABLE:functional_avro_def.alltypesaggmultifilesnopart version: 130
> I0307 21:23:41.012032 21342 catalog-server.cc:477] Publishing update:
> TABLE:functional_avro_def.alltypesaggmultifilesnopart original size: 83
> I0307 21:23:41.012071 21342 CatalogServiceCatalog.java:426] Collected catalog
> update: TABLE:functional_avro_def.alltypesaggnonulls version: 131
> {code}
> We should improve the logging as follows
> * Avoid duplicate messages for the same entity
> * Be clearer about "publishing" and "collecting" a topic update (collect is
> more accurate)
> * Add a log final message when a topic update has been fully assembled
> The double-logging was introduced in IMPALA-5990. For heavily loaded clusters
> this change might lead to a doubling of the log volume - clearly undesirable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-6644) Add last heartbeat timestamp into Statestore metric
Mala Chikka Kempanna created IMPALA-6644: Summary: Add last heartbeat timestamp into Statestore metric Key: IMPALA-6644 URL: https://issues.apache.org/jira/browse/IMPALA-6644 Project: IMPALA Issue Type: Improvement Components: Backend Affects Versions: Impala 2.11.0 Reporter: Mala Chikka Kempanna In the latest and previous versions, statestore in it's default logging reports only when it fails to send heartbeat to any host. There is no way to confirm if Statestore is indeed continuing to heartbeat in all passing conditions, except for turning on debug logs, which becomes too noisy. But at the same time its important to know statestore is indeed heartbeating. The suggestion here is to add a metric in statestore metric page and also print the same in log every once in a minute(or any configurable time-frequency), reporting the last heartbeat timestamp and last heartbeat host(optional). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-6643) Add fine-grained REFRESH privilege
Fredy Wijaya created IMPALA-6643: Summary: Add fine-grained REFRESH privilege Key: IMPALA-6643 URL: https://issues.apache.org/jira/browse/IMPALA-6643 Project: IMPALA Issue Type: Improvement Components: Frontend Reporter: Fredy Wijaya Currently we only need ANY privilege in order to execute INVALIDATE METADATA or REFRESH . We need to have a more fine-grained privilege, i.e. REFRESH to allow executing INVALIDATE METADATA/REFRESH. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-6641) Support more separators between date and time in default timestamp format
Tim Armstrong created IMPALA-6641: - Summary: Support more separators between date and time in default timestamp format Key: IMPALA-6641 URL: https://issues.apache.org/jira/browse/IMPALA-6641 Project: IMPALA Issue Type: Improvement Components: Backend Reporter: Tim Armstrong Following on from IMPALA-5315, we should support further separators between date and time. According to Greg Rahn, "Postgres supports one or more spaces as well as the "T" separator (8601 style 2015-01-01T01:01:01) but more complex things would have to fall under specifying the mask." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-6640) profile collection via standard Impyla usage won't have ExecSummary
Michael Brown created IMPALA-6640:
-
Summary: profile collection via standard Impyla usage won't have
ExecSummary
Key: IMPALA-6640
URL: https://issues.apache.org/jira/browse/IMPALA-6640
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 2.11.0
Reporter: Michael Brown
It seems the Impyla (HS2 client for Python stress test) DB API paradigm doesn't
lend itself to the ideal collection of query profiles.
# It seems a profile will not have an ExecSummary, End Time, or Errors unless
the profile is requested after a CloseOperation RPC. This makes sense to me.
# GetRuntimeProfile RPC must be called before the CloseSession RPC. This makes
sense to me.
# One can't really use standard, public methods in Impyla to do this:
impala.hiveserver2.HiveServer2Cursor().close() handles both the CloseOperation
and CloseSession RPCs in a single call.
# The same is true of the cursor in a context manager \_\_exit\_\_ setting.
\_\_exit\_\_ just calls close()
This means we have to tap into the more private bits of Impyla to do this.
{noformat}
#!/usr/bin/env impala-python
import impala.dbapi
from impala._thrift_api import TGetRuntimeProfileReq
with impala.dbapi.connect(host='localhost', port=21050) as conn:
with conn.cursor() as cursor:
cursor.execute('SELECT COUNT(2) FROM functional.alltypes')
cursor.fetchall()
with open('profile-before-close-operation.txt', 'w') as fh:
fh.write(cursor.get_profile())
# Normal cursor context would end here, but
# Issue CloseOperation RPC without CloseSession RPC.
# This is impala.hiveserver2.Operation().close()
resp = cursor._last_operation.close()
req = TGetRuntimeProfileReq(operationHandle=cursor._last_operation.handle,
sessionHandle=cursor._last_operation.session.handle)
resp = cursor._last_operation._rpc('GetRuntimeProfile', req)
# impala.hiveserver2.HiveServer2Cursor()._reset_state() assist
cursor._last_operation_active = False
with open('profile-after-close-operation.txt', 'w') as fh:
fh.write(resp.profile)
{noformat}
We should think about pythonic ways to make this better for Impyla, or at least
alter the stress test to get the profiles of the queries after CloseOperation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-6639) Crash with 'ORDER BY' in 'OVER' clause with 'RANDOM'
Balazs Jeszenszky created IMPALA-6639:
-
Summary: Crash with 'ORDER BY' in 'OVER' clause with 'RANDOM'
Key: IMPALA-6639
URL: https://issues.apache.org/jira/browse/IMPALA-6639
Project: IMPALA
Issue Type: Bug
Reporter: Balazs Jeszenszky
The following query crashes Impala reliably:
{code:java}
select AVG(n) OVER(ORDER By n) from (
select RANDOM() as n from (select 1 union all select 1) a) b;
{code}
Stack trace:
{code:java}
#0 0x7f98565315e5 in raise () from /lib64/libc.so.6
#1 0x7f9856532dc5 in abort () from /lib64/libc.so.6
#2 0x7f9858697a55 in os::abort(bool) () from
/usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#3 0x7f9858817f87 in VMError::report_and_die() ()
from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#4 0x7f985869c96f in JVM_handle_linux_signal ()
from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#5
#6 0x02c9ed73 in impala::Sorter::Run::Run (this=0x94612a0,
parent=0xa396080, sort_tuple_desc=0x8d9a750,
initial_run=true) at
/usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/sorter.cc:624
#7 0x02ca5c00 in impala::Sorter::Open (this=0xa396080)
at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/sorter.cc:1551
#8 0x02901a09 in impala::SortNode::Open (this=0x945de00,
state=0x9d92180)
at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/sort-node.cc:82
#9 0x02919bcb in impala::AnalyticEvalNode::Open (this=0x6dbb100,
state=0x9d92180)
at
/usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/analytic-eval-node.cc:187
#10 0x01893d03 in impala::FragmentInstanceState::Open (this=0xae64760)
at
/usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:255
#11 0x018917fd in impala::FragmentInstanceState::Exec (this=0xae64760)
at
/usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/fragment-instance-state.cc:80
#12 0x0187a7ac in impala::QueryState::ExecFInstance (this=0x8fd0d00,
fis=0xae64760)
at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/query-state.cc:382
#13 0x0187906e in impala::QueryState::::operator()(void)
const (__closure=0x7f97ffd88bc8)
at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/runtime/query-state.cc:325
#14 0x0187b3eb in
boost::detail::function::void_function_obj_invoker0,
void>::invoke(boost::detail::function::function_buffer &)
(function_obj_ptr=...)
at
/usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
#15 0x017c88de in boost::function0::operator()
(this=0x7f97ffd88bc0)
at
/usr/src/debug/impala-2.11.0-cdh5.14.0/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
#16 0x01abe113 in impala::Thread::SuperviseThread (name=...,
category=..., functor=...,
thread_started=0x7f9800788ab0) at
/usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/util/thread.cc:352
#17 0x01ac6c9e in
boost::_bi::list4 >,
boost::_bi::value >, boost::_bi::value >,
boost::_bi::value >::operator()&, const std::basic_string&,
boost::function, impala::Promise*),
boost::_bi::list0>(boost::_bi::type, void (*&)(const
std::basic_string &, const
std::basic_string &,
boost::function, impala::Promise *), boost::_bi::list0 &, int)
(this=0xa3901c0, f=@0xa3901b8, a=...)
{code}
sort_tuple_size_ ends up as zero in this division:
https://github.com/cloudera/Impala/blob/cdh5-2.11.0_5.14.0/be/src/runtime/sorter.cc#L624
Looks like the tuple descriptor is malformed:
{code:java}
(gdb) p *(impala::TupleDescriptor *) sort_tuple_desc
$4 = {static LLVM_CLASS_NAME = 0x3d649c4 "class.impala::TupleDescriptor", id_ =
8, table_desc_ = 0x0,
byte_size_ = 0, num_null_bytes_ = 0, null_bytes_offset_ = 0,
slots_ = { >> = {
_M_impl = {> =
{<__gnu_cxx::new_allocator> = {}, }, _M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}}, },
string_slots_ = { >> = {
_M_impl = {> =
{<__gnu_cxx::new_allocator> = {}, }, _M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}}, },
collection_slots_ = { >> = {
_M_impl =
