[
https://issues.apache.org/jira/browse/IMPALA-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776603#comment-16776603
]
ASF subversion and git services commented on IMPALA-6816:
---------------------------------------------------------
Commit a29bec1902ffa1ccfb0539a40b874f7650f6b56e in impala's branch
refs/heads/2.x from Tim Armstrong
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a29bec1 ]
IMPALA-6816: minimise calls to GetMinSubscriberTopicVersion()
min_subscriber_topic_version is expensive to compute (requires iterating
over all subscribers to compute) but is only used by one
subscriber/topic pair: Impalads receiving catalog topic updates.
This patch implements a simple fix - only compute it if a subscriber
asks for it. A more complex alternative would be to maintain
a priority queue of subscriber versions, but that didn't seem worth
the the complexity and risk of bugs.
Testing:
Add a statestore test to validate the versions. It looks like we had a
pre-existing test gap for validating min_subscriber_topic_version so
the test is mainly focused on adding that coverage.
Ran core tests with DEBUG and ASAN.
Change-Id: I8ee7cb2355ba1049b9081e0df344ac41aa4ebeb1
Reviewed-on: http://gerrit.cloudera.org:8080/10705
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Statestore spends a lot of time in GetMinSubscriberTopicVersion()
> -----------------------------------------------------------------
>
> Key: IMPALA-6816
> URL: https://issues.apache.org/jira/browse/IMPALA-6816
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
> Affects Versions: Impala 3.0, Impala 2.12.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Labels: admission-control, statestore
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> {noformat}
> Samples: 13K of event 'cycles', Event count (approx.): 1200870513
> 20.23% statestored impalad [.]
> impala::Statestore::GetMinSubscriberTopicVersion(std::string const&,
> std::string*)
> 7.68% statestored [kernel.kallsyms] [k] find_busiest_group
> 3.46% statestored impalad [.]
> impala::Statestore::Subscriber::LastTopicVersionProcessed(std::string const&)
> const
> 3.26% statestored libc-2.12.so [.] __memcmp_sse4_1
> 1.41% statestored [kernel.kallsyms] [k] find_next_bit
> 1.40% statestored [kernel.kallsyms] [k] cpumask_next_and
> 1.21% statestored libpthread-2.12.so [.] pthread_mutex_lock
> 1.04% statestored libc-2.12.so [.] memcpy
> 1.01% statestored [kernel.kallsyms] [k] _spin_lock
> 0.98% statestored impalad [.] 0x000000000088f903
> 0.93% statestored impalad [.] 0x000000000088f8f5
> 0.91% statestored impalad [.] 0x000000000088f8ea
> 0.85% statestored [kernel.kallsyms] [k] ixgbe_xmit_frame_ring
> 0.77% statestored impalad [.] 0x000000000088f8e3
> 0.75% statestored impalad [.] 0x000000000088f900
> 0.75% statestored impalad [.]
> impala::Statestore::IsPrioritizedTopic(std::string const&)
> 0.73% statestored impalad [.] 0x000000000088f8fa
> 0.72% statestored impalad [.] operator new[](unsigned
> long)
> 0.68% statestored [kernel.kallsyms] [k] tcp_recvmsg
> 0.67% statestored impalad [.] 0x000000000088f8fd
> 0.66% statestored impalad [.]
> impala::Statestore::Topic::BuildDelta(std::string const&, long,
> impala::TTopicDelta*)
> 0.61% statestored [kernel.kallsyms] [k] thread_return
> 0.60% statestored impalad [.] 0x000000000088f8f2
> 0.60% statestored libstdc++.so.6 [.]
> std::string::compare(std::string const&) const
> 0.59% statestored impalad [.] 0x000000000088f8e6
> 0.56% statestored impalad [.] 0x000000000088f8ee
> 0.56% statestored libcrypto.so.1.0.1e [.] aesni_encrypt
> 0.55% statestored impalad [.] 0x000000000088f8e0
> 0.55% statestored [kernel.kallsyms] [k] tcp_transmit_skb
> 0.53% statestored [kernel.kallsyms] [k] fget_light
> 0.51% statestored impalad [.] std::_Rb_tree<void*,
> std::pair<void* const, std::shared_ptr<impala::ThriftClientImpl> >,
> std::_Select1st<std::pair<void* const, std::shared_ptr<impala::ThriftCl
> 0.50% statestored impalad [.]
> apache::thrift::transport::TVirtualTransport<apache::thrift::transport::TBufferedTransport,
> apache::thrift::transport::TBufferBase>::readAll_virt(unsigned char*
> 0.50% statestored impalad [.]
> impala::Statestore::DoSubscriberUpdate(impala::Statestore::UpdateKind, int,
> impala::Statestore::ScheduledSubscriberUpdate const&)
> 0.49% statestored libssl.so.1.0.1e [.] tls1_enc
> 0.48% statestored libssl.so.1.0.1e [.] ssl3_read_bytes
> {noformat}
> We are spending most of our time computing this for non-catalog topics, where
> it's not even used.
> There are a couple of ways we could fix this that I can think of:
> * Avoid including this information for topics where we're not interested in it
> * Cache or precompute the value somehow to avoid iterating over all
> subscribers every time
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
