[jira] [Commented] (HDFS-8213) DFSClient should not instantiate SpanReceiverHost
[
https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509472#comment-14509472
]
Colin Patrick McCabe commented on HDFS-8213:
bq. can you people suggest configuration for DFSClient..?
I'm thinking {{hdfs.client.htrace.spanreceiver.classes}}. It's not completely
trivial because I have to change our SpanReceiverHost thing, but shouldn't be
too bad... let me see if I can post the patch
> DFSClient should not instantiate SpanReceiverHost
> -
>
> Key: HDFS-8213
> URL: https://issues.apache.org/jira/browse/HDFS-8213
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Billie Rinaldi
>Assignee: Brahma Reddy Battula
>Priority: Critical
>
> DFSClient initializing SpanReceivers is a problem for Accumulo, which manages
> SpanReceivers through its own configuration. This results in the same
> receivers being registered multiple times and spans being delivered more than
> once. The documentation says SpanReceiverHost.getInstance should be issued
> once per process, so there is no expectation that DFSClient should do this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8213) DFSClient should not instantiate SpanReceiverHost
[ https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508544#comment-14508544 ] Brahma Reddy Battula commented on HDFS-8213: Thanks [~billie.rinaldi] for raising issue and [~cmccabe] for your inputs .. can you people suggest configuration for DFSClient..? > DFSClient should not instantiate SpanReceiverHost > - > > Key: HDFS-8213 > URL: https://issues.apache.org/jira/browse/HDFS-8213 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Billie Rinaldi >Assignee: Brahma Reddy Battula >Priority: Critical > > DFSClient initializing SpanReceivers is a problem for Accumulo, which manages > SpanReceivers through its own configuration. This results in the same > receivers being registered multiple times and spans being delivered more than > once. The documentation says SpanReceiverHost.getInstance should be issued > once per process, so there is no expectation that DFSClient should do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8213) DFSClient should not instantiate SpanReceiverHost
[ https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508190#comment-14508190 ] Billie Rinaldi commented on HDFS-8213: -- I agree, having DFSClient use a separate configuration key seems like it would satisfy all the issues we've discussed. Thanks for coming up with a solution that works for both of us, [~cmccabe]. > DFSClient should not instantiate SpanReceiverHost > - > > Key: HDFS-8213 > URL: https://issues.apache.org/jira/browse/HDFS-8213 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Billie Rinaldi >Assignee: Brahma Reddy Battula >Priority: Critical > > DFSClient initializing SpanReceivers is a problem for Accumulo, which manages > SpanReceivers through its own configuration. This results in the same > receivers being registered multiple times and spans being delivered more than > once. The documentation says SpanReceiverHost.getInstance should be issued > once per process, so there is no expectation that DFSClient should do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8213) DFSClient should not instantiate SpanReceiverHost
[
https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508049#comment-14508049
]
Colin Patrick McCabe commented on HDFS-8213:
bq. Yes, clients need tracing, and when they do they should enable it
themselves. FsShell should enable tracing when it wants to use it, instead of
doing that in DFSClient.
There are hundreds or maybe even thousands of programs that use the HDFS
client. It's not practical to modify them all to run
{{Trace#addSpanReceiver}}. In some cases the programs that use HDFS are even
proprietary or customer programs where we don't have access to the source code.
I have some ideas for how to make this all work better with a better interface
in {{Tracer}}. We might need an incompatible interface change to do it,
though. For now, let's just change the config key for DFSClient... that should
fix the problem for Accumulo.
> DFSClient should not instantiate SpanReceiverHost
> -
>
> Key: HDFS-8213
> URL: https://issues.apache.org/jira/browse/HDFS-8213
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Billie Rinaldi
>Assignee: Brahma Reddy Battula
>Priority: Critical
>
> DFSClient initializing SpanReceivers is a problem for Accumulo, which manages
> SpanReceivers through its own configuration. This results in the same
> receivers being registered multiple times and spans being delivered more than
> once. The documentation says SpanReceiverHost.getInstance should be issued
> once per process, so there is no expectation that DFSClient should do this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8213) DFSClient should not instantiate SpanReceiverHost
[
https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507888#comment-14507888
]
Colin Patrick McCabe commented on HDFS-8213:
bq. I think Billie Rinaldi is correct here; the client should not instantiate
it's own SpanReceiverHost, but instead depend on the process in which it
resides to provide. This is how HBase client works as well.
[~ndimiduk], what if I want to trace an HBase PUT all the way through the
system? You're saying that the HBase client can't activate tracing on its own,
so I have to make code changes to the process doing the PUT (i.e. the user of
the HBase client) in order to get that info? Seems like a limitation.
It's also worth pointing out that adding a {{SpanReceiverHost}} to the
{{DFSClient}} is not really a new change... it goes back to HDFS-7055, last
October. So it's been in there at least 6 months. Of course we can revisit it
if that makes sense, but it's not really "new" except in the sense that it took
a very long time to do another Hadoop release with that in it. (We really
should start being better with releases...)
Thinking about this a little more, another possible resolution here is to
change the configuration keys which the DFSClient looks for so that it's
different than the ones which the NameNode and DataNode look for. Right now
{{hadoop.htrace.spanreceiver.classes}} will activate span receivers in both the
NN and the DFSClient. But the DFSClient could instead look for
{{hdfs.client.htrace.spanreceiver.classes}}. Then [~billie.rinaldi] could use
the same configuration file for everything, and the dfsclient would never
create its own span receivers or samplers. And I could continue to trace the
dfsclient without modifying daemon code. Seems like a good resolution.
> DFSClient should not instantiate SpanReceiverHost
> -
>
> Key: HDFS-8213
> URL: https://issues.apache.org/jira/browse/HDFS-8213
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Billie Rinaldi
>Assignee: Brahma Reddy Battula
>Priority: Critical
>
> DFSClient initializing SpanReceivers is a problem for Accumulo, which manages
> SpanReceivers through its own configuration. This results in the same
> receivers being registered multiple times and spans being delivered more than
> once. The documentation says SpanReceiverHost.getInstance should be issued
> once per process, so there is no expectation that DFSClient should do this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8213) DFSClient should not instantiate SpanReceiverHost
[ https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507847#comment-14507847 ] Billie Rinaldi commented on HDFS-8213: -- Yes, clients need tracing, and when they do they should enable it themselves. FsShell should enable tracing when it wants to use it, instead of doing that in DFSClient. A solution in 2.7.1 would be sufficient. I have concerns about bumping the htrace dependency version as htrace 3.2 has many API incompatibilities with 3.1. I've begun to compile them on ACCUMULO-3741. > DFSClient should not instantiate SpanReceiverHost > - > > Key: HDFS-8213 > URL: https://issues.apache.org/jira/browse/HDFS-8213 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Billie Rinaldi >Assignee: Brahma Reddy Battula >Priority: Critical > > DFSClient initializing SpanReceivers is a problem for Accumulo, which manages > SpanReceivers through its own configuration. This results in the same > receivers being registered multiple times and spans being delivered more than > once. The documentation says SpanReceiverHost.getInstance should be issued > once per process, so there is no expectation that DFSClient should do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8213) DFSClient should not instantiate SpanReceiverHost
[
https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507819#comment-14507819
]
Colin Patrick McCabe commented on HDFS-8213:
bq. The hadoop.htrace.span.receiver.classes is not set in Accumulo
configuration files, but it is set in Hadoop configuration files. Accumulo uses
Hadoop configuration files to connect to HDFS, thus its uses of DFSClient will
have Hadoop's hadoop.htrace.span.receiver.classes. HBase does something
similar, I believe.
The way Cloudera Manager manages configuration files is that it creates
separate config files for each daemon. So the NameNode reads its own set of
config files, the DataNode reads a separate set, Hive reads another set, Flume
reads still another set, etc. etc. So {{hadoop.htrace.span.receiver.classes}}
would be set in the NN and DN configuration files, but not in the ones
targetted towards the DFSClients. Does Ambari do something similar? It seems
like using the same set of configuration files for everything would be very
limiting, if you wanted to do something like turn on short circuit for some
clients but not for others, etc.
I know from a developer perspective it's frustrating to not be able to use the
same config files for every daemon (I like to do that myself) but it's not
broken, just inconvenient.
bq. No. The way it works (did work, until this change was introduced in
DFSClient) is that server processes instantiate SpanReceiverHost. If an app
wants tracing, it also has to instantiate SpanReceiverHost. The Accumulo client
does not instantiate SPH itself, as DFSClient should not.
It's not true that only server processes need tracing. Clients also need
tracing. For example, one test I do a lot is to run FsShell with tracing
turned on. This would not be possible if only servers had tracing. The point
that I was making with my example is that the Accumulo client itself probably
should have tracing too, and this would potentially conflict with another
server using the Accumulo client.
bq. The change in DFSClient changes how apps are supposed to use tracing. It
seems like this would be mitigated by deduping SpanReceivers in htrace, but if
we go that route I would like the DFSClient change to be reverted until HDFS
moves to a version of htrace with deduping. Otherwise, Accumulo and HBase will
have to leave HDFS tracing disabled, or change how they're configuring HDFS, if
they wish to avoid double delivery of spans.
We're doing a new release of HTrace soon... like this week or the next. If we
can get the deduping into the 3.2 release, we can bump the version in Hadoop
2.7.1. We can't change what's in Hadoop 2.7.0, that release is done.
Thanks again for trying this stuff out. I'm going to work on a deduping patch
for HTrace, would appreciate a review.
> DFSClient should not instantiate SpanReceiverHost
> -
>
> Key: HDFS-8213
> URL: https://issues.apache.org/jira/browse/HDFS-8213
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Billie Rinaldi
>Assignee: Brahma Reddy Battula
>Priority: Critical
>
> DFSClient initializing SpanReceivers is a problem for Accumulo, which manages
> SpanReceivers through its own configuration. This results in the same
> receivers being registered multiple times and spans being delivered more than
> once. The documentation says SpanReceiverHost.getInstance should be issued
> once per process, so there is no expectation that DFSClient should do this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8213) DFSClient should not instantiate SpanReceiverHost
[ https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507771#comment-14507771 ] Billie Rinaldi commented on HDFS-8213: -- The hadoop.htrace.span.receiver.classes is not set in Accumulo configuration files, but it is set in Hadoop configuration files. Accumulo uses Hadoop configuration files to connect to HDFS, thus its uses of DFSClient will have Hadoop's hadoop.htrace.span.receiver.classes. HBase does something similar, I believe. bq. Plus, it just kicks the problem up to a higher level. If my FooProcess wants to use both HTrace and Accumulo, FooProcess could easily make the same argument that "Accumulo should not instantiate SpanReceiverHost" since FooProcess is already doing that. And since FooProcess uses the accumulo client, it would conflict with whatever accumulo was configuring, if the same config file was used for everything. No. The way it works (did work, until this change was introduced in DFSClient) is that server processes instantiate SpanReceiverHost. If an app wants tracing, it also has to instantiate SpanReceiverHost. The Accumulo client does not instantiate SPH itself, as DFSClient should not. bq. One thing we could do to make this a little less painful is to deduplicate span receivers inside the library. So if both DFSClient and Accumlo requested an HTracedSpanReceiver, we could simply create one instance of that. This would allow us to use the same config file for everything. The change in DFSClient changes how apps are supposed to use tracing. It seems like this would be mitigated by deduping SpanReceivers in htrace, but if we go that route I would like the DFSClient change to be reverted until HDFS moves to a version of htrace with deduping. Otherwise, Accumulo and HBase will have to leave HDFS tracing disabled, or change how they're configuring HDFS, if they wish to avoid double delivery of spans. > DFSClient should not instantiate SpanReceiverHost > - > > Key: HDFS-8213 > URL: https://issues.apache.org/jira/browse/HDFS-8213 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Billie Rinaldi >Assignee: Brahma Reddy Battula >Priority: Critical > > DFSClient initializing SpanReceivers is a problem for Accumulo, which manages > SpanReceivers through its own configuration. This results in the same > receivers being registered multiple times and spans being delivered more than > once. The documentation says SpanReceiverHost.getInstance should be issued > once per process, so there is no expectation that DFSClient should do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8213) DFSClient should not instantiate SpanReceiverHost
[
https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507643#comment-14507643
]
Colin Patrick McCabe commented on HDFS-8213:
Thanks again for kicking the tires on htrace, [~billie.rinaldi]. Let me see if
I can get to the bottom of this.
bq. As documented, each process must configure its own span receivers if it
wants to use tracing. If I set hadoop.htrace.span.receiver.classes to the empty
string, then the NameNode and DataNode will not do any tracing.
You are right that you need to set {{hadoop.htrace.span.receiver.classes}} in
the NameNode and DataNode configuration. However, you need to avoid setting it
in the Accumulo configuration... instead, use whatever configuration Accumulo
uses to set this value. This means that you can't use the same config file for
the NN and DN as for the DFSClient, currently.
bq. If span receiver initialization in DFSClient is important to the use of the
hadoop.htrace.sampler configuration property, perhaps a compromise would be to
perform SpanReceiverHost.getInstance only when the sampler is set to something
other than NeverSampler.
Keep in mind that {{hadoop.htrace.sampler}} is a completely different
configuration key than {{hadoop.htrace.span.receiver.classes}}. If you are
sampling at the level of Accumulo operations, I would not recommend setting
{{hadoop.htrace.sampler}}, in any config file on the cluster. You want all of
the sampling to happen inside accumulo.
bq. I think Billie Rinaldi is correct here; the client should not instantiate
it's own SpanReceiverHost, but instead depend on the process in which it
resides to provide. This is how HBase client works as well.
HBase is exactly the same. In the case of HBase, you do not want to set
{{hadoop.htrace.span.receiver.classes}} in the HBase config files. Instead,
you would set {{hbase.htrace.span.receiver.classes}}. Then HBase would create
a span receiver, and DFSClient would not.
It seems like there is a hidden assumption here that you want to use the same
config file for everything. But we really don't support that right now.
Getting rid of the SpanReceiverHost in DFSClient is not an option since some
people want to just trace HDFS without tracing any other system. Plus, it just
kicks the problem up to a higher level. If my FooProcess wants to use both
HTrace and Accumulo, FooProcess could easily make the same argument that
"Accumulo should not instantiate SpanReceiverHost" since FooProcess is already
doing that. And since FooProcess uses the accumulo client, it would conflict
with whatever accumulo was configuring, if the same config file was used for
everything.
One thing we could do to make this a little less painful is to deduplicate span
receivers inside the library. So if both DFSClient and Accumlo requested an
HTracedSpanReceiver, we could simply create one instance of that. This would
allow us to use the same config file for everything.
As a side note, [~billie.rinaldi], can you explain how you configure which
sampler and span receiver accumulo uses? In HBase we set it to
{{hbase.htrace.span.receiver.classes}}, etc. I would recommend something like
{{accumulo.htrace.span.receiver.classes}} for consistency. This also allows
you to sue the same config file for everything since it doesn't conflict with
the keys which Hadoop uses to set these values. That is the reason why we set
up the "hbase.htrace" "namespace" as separate from the "hadoop.htrace"
"namespace" if you see what I'm saying.
> DFSClient should not instantiate SpanReceiverHost
> -
>
> Key: HDFS-8213
> URL: https://issues.apache.org/jira/browse/HDFS-8213
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Billie Rinaldi
>Assignee: Brahma Reddy Battula
>Priority: Critical
>
> DFSClient initializing SpanReceivers is a problem for Accumulo, which manages
> SpanReceivers through its own configuration. This results in the same
> receivers being registered multiple times and spans being delivered more than
> once. The documentation says SpanReceiverHost.getInstance should be issued
> once per process, so there is no expectation that DFSClient should do this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8213) DFSClient should not instantiate SpanReceiverHost
[ https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505860#comment-14505860 ] Nick Dimiduk commented on HDFS-8213: I think [~billie.rinaldi] is correct here; the client should not instantiate it's own SpanReceiverHost, but instead depend on the process in which it resides to provide. This is how HBase client works as well. > DFSClient should not instantiate SpanReceiverHost > - > > Key: HDFS-8213 > URL: https://issues.apache.org/jira/browse/HDFS-8213 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Billie Rinaldi >Priority: Critical > > DFSClient initializing SpanReceivers is a problem for Accumulo, which manages > SpanReceivers through its own configuration. This results in the same > receivers being registered multiple times and spans being delivered more than > once. The documentation says SpanReceiverHost.getInstance should be issued > once per process, so there is no expectation that DFSClient should do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8213) DFSClient should not instantiate SpanReceiverHost
[ https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505854#comment-14505854 ] Billie Rinaldi commented on HDFS-8213: -- If span receiver initialization in DFSClient is important to the use of the hadoop.htrace.sampler configuration property, perhaps a compromise would be to perform SpanReceiverHost.getInstance only when the sampler is set to something other than NeverSampler. > DFSClient should not instantiate SpanReceiverHost > - > > Key: HDFS-8213 > URL: https://issues.apache.org/jira/browse/HDFS-8213 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Billie Rinaldi >Priority: Critical > > DFSClient initializing SpanReceivers is a problem for Accumulo, which manages > SpanReceivers through its own configuration. This results in the same > receivers being registered multiple times and spans being delivered more than > once. The documentation says SpanReceiverHost.getInstance should be issued > once per process, so there is no expectation that DFSClient should do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8213) DFSClient should not instantiate SpanReceiverHost
[ https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505847#comment-14505847 ] Billie Rinaldi commented on HDFS-8213: -- As documented, each process must configure its own span receivers if it wants to use tracing. If I set hadoop.htrace.span.receiver.classes to the empty string, then the NameNode and DataNode will not do any tracing. > DFSClient should not instantiate SpanReceiverHost > - > > Key: HDFS-8213 > URL: https://issues.apache.org/jira/browse/HDFS-8213 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Billie Rinaldi >Priority: Critical > > DFSClient initializing SpanReceivers is a problem for Accumulo, which manages > SpanReceivers through its own configuration. This results in the same > receivers being registered multiple times and spans being delivered more than > once. The documentation says SpanReceiverHost.getInstance should be issued > once per process, so there is no expectation that DFSClient should do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8213) DFSClient should not instantiate SpanReceiverHost
[
https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505797#comment-14505797
]
Colin Patrick McCabe commented on HDFS-8213:
Hi Billie,
{{DFSClient}} needs to instantiate {{SpanReceiverHost}} in order to implement
tracing, in the case where the process using the {{DFSClient}} doesn't
configure its own span receivers.
If you are concerned about multiple span receivers being instantiated, simply
set {{hadoop.htrace.span.receiver.classes}} to the empty string, and Hadoop
won't instantiate any span receivers. That should be its default anyway.
> DFSClient should not instantiate SpanReceiverHost
> -
>
> Key: HDFS-8213
> URL: https://issues.apache.org/jira/browse/HDFS-8213
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Billie Rinaldi
>Priority: Critical
>
> DFSClient initializing SpanReceivers is a problem for Accumulo, which manages
> SpanReceivers through its own configuration. This results in the same
> receivers being registered multiple times and spans being delivered more than
> once. The documentation says SpanReceiverHost.getInstance should be issued
> once per process, so there is no expectation that DFSClient should do this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
