[
https://issues.apache.org/jira/browse/HDFS-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507819#comment-14507819
]
Colin Patrick McCabe commented on HDFS-8213:
--------------------------------------------
bq. The hadoop.htrace.span.receiver.classes is not set in Accumulo
configuration files, but it is set in Hadoop configuration files. Accumulo uses
Hadoop configuration files to connect to HDFS, thus its uses of DFSClient will
have Hadoop's hadoop.htrace.span.receiver.classes. HBase does something
similar, I believe.
The way Cloudera Manager manages configuration files is that it creates
separate config files for each daemon. So the NameNode reads its own set of
config files, the DataNode reads a separate set, Hive reads another set, Flume
reads still another set, etc. etc. So {{hadoop.htrace.span.receiver.classes}}
would be set in the NN and DN configuration files, but not in the ones
targetted towards the DFSClients. Does Ambari do something similar? It seems
like using the same set of configuration files for everything would be very
limiting, if you wanted to do something like turn on short circuit for some
clients but not for others, etc.
I know from a developer perspective it's frustrating to not be able to use the
same config files for every daemon (I like to do that myself) but it's not
broken, just inconvenient.
bq. No. The way it works (did work, until this change was introduced in
DFSClient) is that server processes instantiate SpanReceiverHost. If an app
wants tracing, it also has to instantiate SpanReceiverHost. The Accumulo client
does not instantiate SPH itself, as DFSClient should not.
It's not true that only server processes need tracing. Clients also need
tracing. For example, one test I do a lot is to run FsShell with tracing
turned on. This would not be possible if only servers had tracing. The point
that I was making with my example is that the Accumulo client itself probably
should have tracing too, and this would potentially conflict with another
server using the Accumulo client.
bq. The change in DFSClient changes how apps are supposed to use tracing. It
seems like this would be mitigated by deduping SpanReceivers in htrace, but if
we go that route I would like the DFSClient change to be reverted until HDFS
moves to a version of htrace with deduping. Otherwise, Accumulo and HBase will
have to leave HDFS tracing disabled, or change how they're configuring HDFS, if
they wish to avoid double delivery of spans.
We're doing a new release of HTrace soon... like this week or the next. If we
can get the deduping into the 3.2 release, we can bump the version in Hadoop
2.7.1. We can't change what's in Hadoop 2.7.0, that release is done.
Thanks again for trying this stuff out. I'm going to work on a deduping patch
for HTrace, would appreciate a review.
> DFSClient should not instantiate SpanReceiverHost
> -------------------------------------------------
>
> Key: HDFS-8213
> URL: https://issues.apache.org/jira/browse/HDFS-8213
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.7.0
> Reporter: Billie Rinaldi
> Assignee: Brahma Reddy Battula
> Priority: Critical
>
> DFSClient initializing SpanReceivers is a problem for Accumulo, which manages
> SpanReceivers through its own configuration. This results in the same
> receivers being registered multiple times and spans being delivered more than
> once. The documentation says SpanReceiverHost.getInstance should be issued
> once per process, so there is no expectation that DFSClient should do this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
