[jira] [Commented] (HBASE-26366) ZK interaction during Master startup produces loads of root-less spans
[
https://issues.apache.org/jira/browse/HBASE-26366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17539463#comment-17539463
]
Nick Dimiduk commented on HBASE-26366:
--
In fact this is not just Master startup, but Region Server startup as well.
> ZK interaction during Master startup produces loads of root-less spans
> --
>
> Key: HBASE-26366
> URL: https://issues.apache.org/jira/browse/HBASE-26366
> Project: HBase
> Issue Type: Improvement
> Components: tracing
>Affects Versions: 2.5.0, 3.0.0-alpha-2
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
> Attachments: image.png,
> regroup-spans-for-recoverablezookeeper.create.png
>
>
> Enable tracing with {{-Dotel.traces.sampler=always_on}} and launch the
> standalone master, check the trace store. We see lots of single-span traces,
> each a call to different {{RecoverableZooKeeper}} methods, like {{create}}
> and {{exists}}. Whatever master startup thread this is should define its own
> root span.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26366) ZK interaction during Master startup produces loads of root-less spans
[
https://issues.apache.org/jira/browse/HBASE-26366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533314#comment-17533314
]
Andrew Kyle Purtell commented on HBASE-26366:
-
I scheduled fix versions for branch-2 and master. We can add the appropriate
fix version for 2.5 when it lands.
> ZK interaction during Master startup produces loads of root-less spans
> --
>
> Key: HBASE-26366
> URL: https://issues.apache.org/jira/browse/HBASE-26366
> Project: HBase
> Issue Type: Improvement
> Components: tracing
>Affects Versions: 2.5.0, 3.0.0-alpha-2
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
> Attachments: image.png,
> regroup-spans-for-recoverablezookeeper.create.png
>
>
> Enable tracing with {{-Dotel.traces.sampler=always_on}} and launch the
> standalone master, check the trace store. We see lots of single-span traces,
> each a call to different {{RecoverableZooKeeper}} methods, like {{create}}
> and {{exists}}. Whatever master startup thread this is should define its own
> root span.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26366) ZK interaction during Master startup produces loads of root-less spans
[
https://issues.apache.org/jira/browse/HBASE-26366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531152#comment-17531152
]
Nick Dimiduk commented on HBASE-26366:
--
I have a patch started but it's not ready for review. I think it's okay if this
lands as a follow-on in 2.5.1. Thanks [~apurtell].
> ZK interaction during Master startup produces loads of root-less spans
> --
>
> Key: HBASE-26366
> URL: https://issues.apache.org/jira/browse/HBASE-26366
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0, 3.0.0-alpha-2
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
> Attachments: image.png,
> regroup-spans-for-recoverablezookeeper.create.png
>
>
> Enable tracing with {{-Dotel.traces.sampler=always_on}} and launch the
> standalone master, check the trace store. We see lots of single-span traces,
> each a call to different {{RecoverableZooKeeper}} methods, like {{create}}
> and {{exists}}. Whatever master startup thread this is should define its own
> root span.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26366) ZK interaction during Master startup produces loads of root-less spans
[
https://issues.apache.org/jira/browse/HBASE-26366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530990#comment-17530990
]
Andrew Kyle Purtell commented on HBASE-26366:
-
Going to roll 2.5.0RC0 soon. Keep this in for imminent resolution or bump it?
> ZK interaction during Master startup produces loads of root-less spans
> --
>
> Key: HBASE-26366
> URL: https://issues.apache.org/jira/browse/HBASE-26366
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0, 3.0.0-alpha-2
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
> Attachments: image.png,
> regroup-spans-for-recoverablezookeeper.create.png
>
>
> Enable tracing with {{-Dotel.traces.sampler=always_on}} and launch the
> standalone master, check the trace store. We see lots of single-span traces,
> each a call to different {{RecoverableZooKeeper}} methods, like {{create}}
> and {{exists}}. Whatever master startup thread this is should define its own
> root span.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26366) ZK interaction during Master startup produces loads of root-less spans
[
https://issues.apache.org/jira/browse/HBASE-26366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17502248#comment-17502248
]
Nick Dimiduk commented on HBASE-26366:
--
I do intend to resolve this as part of HBASE-26419. Let me make it a subtask of
that one.
> ZK interaction during Master startup produces loads of root-less spans
> --
>
> Key: HBASE-26366
> URL: https://issues.apache.org/jira/browse/HBASE-26366
> Project: HBase
> Issue Type: Bug
> Components: tracing
>Affects Versions: 2.5.0, 3.0.0-alpha-2
>Reporter: Nick Dimiduk
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
> Attachments: image.png,
> regroup-spans-for-recoverablezookeeper.create.png
>
>
> Enable tracing with {{-Dotel.traces.sampler=always_on}} and launch the
> standalone master, check the trace store. We see lots of single-span traces,
> each a call to different {{RecoverableZooKeeper}} methods, like {{create}}
> and {{exists}}. Whatever master startup thread this is should define its own
> root span.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26366) ZK interaction during Master startup produces loads of root-less spans
[
https://issues.apache.org/jira/browse/HBASE-26366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501526#comment-17501526
]
Andrew Kyle Purtell commented on HBASE-26366:
-
What are the chances of wrapping this work up this month to get out 2.5.0?
[~ndimiduk] I might cut a 2.5.0-alpha now at least to move that forward even
though this work would not be complete until a 2.5.0 release later, otherwise.
> ZK interaction during Master startup produces loads of root-less spans
> --
>
> Key: HBASE-26366
> URL: https://issues.apache.org/jira/browse/HBASE-26366
> Project: HBase
> Issue Type: Bug
> Components: tracing
>Affects Versions: 2.5.0, 3.0.0-alpha-2
>Reporter: Nick Dimiduk
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
> Attachments: image.png,
> regroup-spans-for-recoverablezookeeper.create.png
>
>
> Enable tracing with {{-Dotel.traces.sampler=always_on}} and launch the
> standalone master, check the trace store. We see lots of single-span traces,
> each a call to different {{RecoverableZooKeeper}} methods, like {{create}}
> and {{exists}}. Whatever master startup thread this is should define its own
> root span.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26366) ZK interaction during Master startup produces loads of root-less spans
[
https://issues.apache.org/jira/browse/HBASE-26366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438834#comment-17438834
]
Nick Dimiduk commented on HBASE-26366:
--
There's an argument to be made for remove tracing from this part of the code.
These ZK operations are not in the critical path of user data queries, so I
think the odds of an operator having interest in this is low.
The counter argument is that when this gets stuck, having traces could be
really helpful. {{blockUntilBecomingActive}} is itself too long-lived to be the
root span, but maybe there's a point when is starts to take over ownership that
makes sense to be the start of a span, and one that we emit 100% of the time.
> ZK interaction during Master startup produces loads of root-less spans
> --
>
> Key: HBASE-26366
> URL: https://issues.apache.org/jira/browse/HBASE-26366
> Project: HBase
> Issue Type: Bug
> Components: tracing
>Affects Versions: 2.5.0, 3.0.0-alpha-2
>Reporter: Nick Dimiduk
>Priority: Major
> Attachments: image.png,
> regroup-spans-for-recoverablezookeeper.create.png
>
>
> Enable tracing with {{-Dotel.traces.sampler=always_on}} and launch the
> standalone master, check the trace store. We see lots of single-span traces,
> each a call to different {{RecoverableZooKeeper}} methods, like {{create}}
> and {{exists}}. Whatever master startup thread this is should define its own
> root span.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26366) ZK interaction during Master startup produces loads of root-less spans
[
https://issues.apache.org/jira/browse/HBASE-26366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438454#comment-17438454
]
Tak-Lon (Stephen) Wu commented on HBASE-26366:
--
I found the potential issue of this single span(s), basically it's due to the
static class ZKUtil calls {{zkw.getRecoverableZooKeeper().create}} whenever a
znode or znode with its parent (normally /hbase) need to be created, e.g.
{{ZKUtil.createAndWatch}}, {{ZKUtil.createWithParents}}, {{ZKUtil.getData}}
If we're going to group all these spans, we need to wrap a parent span at their
parent level, e.g. I found at few places, 1/ ClientZKSyncer, 2/ Procedure /
ProcedureMember, 3/ HMaster#startActiveMasterManager (this is a private method)
and etc.
if this is the direction, the scope of this JIRAs are big and may need to
create more sub-tasks.
Moreover, based few experiment I did to group spans for
HMaster#startActiveMasterManager, IMO we should replace some current code of
{{Span span = TraceUtil.getGlobalTracer()}}or {{TraceUtil.trace()}} with
[{{@WithSpan}}
annotation|https://github.com/open-telemetry/opentelemetry-java/blob/main/extensions/annotations/src/main/java/io/opentelemetry/extension/annotations/WithSpan.java]
(see the [code
example|https://github.com/open-telemetry/opentelemetry-java/blob/main/extensions/annotations/src/test/java/io/opentelemetry/extension/annotations/WithSpanUsageExamples.java]
and don't hack too much code in TraceUtils)
> ZK interaction during Master startup produces loads of root-less spans
> --
>
> Key: HBASE-26366
> URL: https://issues.apache.org/jira/browse/HBASE-26366
> Project: HBase
> Issue Type: Bug
> Components: tracing
>Affects Versions: 2.5.0, 3.0.0-alpha-2
>Reporter: Nick Dimiduk
>Priority: Major
> Attachments: image.png
>
>
> Enable tracing with {{-Dotel.traces.sampler=always_on}} and launch the
> standalone master, check the trace store. We see lots of single-span traces,
> each a call to different {{RecoverableZooKeeper}} methods, like {{create}}
> and {{exists}}. Whatever master startup thread this is should define its own
> root span.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26366) ZK interaction during Master startup produces loads of root-less spans
[
https://issues.apache.org/jira/browse/HBASE-26366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430249#comment-17430249
]
Nick Dimiduk commented on HBASE-26366:
--
!image.png!
> ZK interaction during Master startup produces loads of root-less spans
> --
>
> Key: HBASE-26366
> URL: https://issues.apache.org/jira/browse/HBASE-26366
> Project: HBase
> Issue Type: Bug
> Components: tracing
>Affects Versions: 2.5.0, 3.0.0-alpha-2
>Reporter: Nick Dimiduk
>Priority: Major
> Attachments: image.png
>
>
> Enable tracing with {{-Dotel.traces.sampler=always_on}} and launch the
> standalone master, check the trace store. We see lots of single-span traces,
> each a call to different {{RecoverableZooKeeper}} methods, like {{create}}
> and {{exists}}. Whatever master startup thread this is should define its own
> root span.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
