Thanks Dave,
Yes, we are using Accumulo 2.0.1.
The workaround `--address accumulo_tracer`works perfectly in my local
development environment.
We can see compaction trace messages in the trace table.
Thanks again Christopher!
We are deploying in AWS to do final testing...
Cheers!...
...Keith
On 2022/07/29 19:09:00 Christopher wrote:
> If the version being used is 2.0.1, then that version is still
affected by
> the bug you mention. However, since TraceServer never used
AbstractServer,
> it was not affected and should still work fine.
>
> On Fri, Jul 29, 2022 at 3:05 PM Dave Marion <dm...@gmail.com> wrote:
>
> > Ok, I wasn't sure what version was being used here. I saw a
reference to
> > 2.0.1 in an earlier email.
> >
> > On Fri, Jul 29, 2022 at 3:03 PM Christopher <ct...@apache.org> wrote:
> >
> > > That was a bug in the new AbstractServer class in 2.x. I don't
think it
> > > ever affected 1.x
> > > I checked the 1.10 code and it wouldn't affect the tracer server.
`-a`
> > > should still work fine there.
> > >
> > > On Fri, Jul 29, 2022 at 3:00 PM Dave Marion <dm...@gmail.com> wrote:
> > >
> > > > https://github.com/apache/accumulo/pull/2119 fixed a bug in 2.1
where
> > > the
> > > > -a argument was not correctly setting the hostname. It looks
like 2.0.1
> > > is
> > > > affected by this too.
> > > >
> > > > On Fri, Jul 29, 2022 at 12:57 PM Christopher <ct...@apache.org>
> > > wrote:
> > > >
> > > > > The tracer should be advertising its own address in ZK. By
default,
> > the
> > > > > server listens on `0.0.0.0`, unless `-a` or `--address` is
specified
> > on
> > > > the
> > > > > command-line when it is started. Most server types use a utility
> > class
> > > > that
> > > > > will use `InetAddress.getLocalHost().getCanonicalHostName()`
for the
> > > > > advertisement address if it sees that it is listening on
`0.0.0.0`.
> > > > > However, it looks like the tracer doesn't do this.
> > > > >
> > > > > I would try to verify that `0.0.0.0` actually appears as the
> > advertised
> > > > > address in ZK for the tracer service, just be sure it's not a
> > > > > ZooTraceClient bug on the tablet server side (but I'm pretty sure
> > it's
> > > > not,
> > > > > after looking at the code).
> > > > >
> > > > > As a workaround, I would start up the tracer service using a
script
> > > that
> > > > > sets `--address accumulo_tracer` on the command-line, where
> > > > > "accumulo_tracer" is the hostname or IP address that you want the
> > > tracer
> > > > > listening on and advertising to other servers. This address
should be
> > > > > bindable where the tracer is running, and reachable from
wherever the
> > > > > tservers are (so make sure it's not a private address only
available
> > > > inside
> > > > > the container running the tracer). The reason for this is
that the
> > > > hostname
> > > > > will resolve to an IP address inside the tracer process, and
then the
> > > > > tracer will advertise the IP address. It won't advertise the
original
> > > > > hostname, if you specified a hostname.
> > > > >
> > > > > I hope that helps.
> > > > >
> > > > >
> > > > > On Fri, Jul 29, 2022 at 12:18 PM kma <km...@phemi.com> wrote:
> > > > >
> > > > > > Nice,
> > > > > >
> > > > > > Turn on debug really help.
> > > > > >
> > > > > > 2022-07-29 12:12:14,235 [tracer.ZooTraceClient] DEBUG: Scanning
> > trace
> > > > > > hosts in zookeeper: /tracers
> > > > > > 2022-07-29 12:12:14,240 [tracer.ZooTraceClient] DEBUG:
Trace hosts:
> > > > > > [0.0.0.0:12234, 0.0.0.0:12234]
> > > > > > 2022-07-29 12:12:14,240 [tracer.ZooTraceClient] DEBUG:
Successfully
> > > > > > initialized tracer hosts from ZooKeeper
> > > > > >
> > > > > > [root@accumulo_gc /]# curl 0.0.0.0:12234
> > > > > > curl: (7) Failed connect to 0.0.0.0:12234; Connection refused
> > > > > >
> > > > > > [root@accumulo_gc /]# curl accumulo_tracer:12234
> > > > > > curl: (52) Empty reply from server
> > > > > >
> > > > > > Is there a way to set Trace hosts to accumulo_tracer:12234
> > > > > >
> > > > > > Cheers!...
> > > > > > ...Keith
> > > > > >
> > > > > >
> > > > > > On 2022/07/28 23:40:18 Christopher wrote:
> > > > > > > Do all servers have the same configuration?
> > > > > > >
> > > > > > > I would investigate the tablet server debug logs to
determine if
> > > > it's
> > > > > > > having trouble setting up tracing. It should be able to
locate
> > the
> > > > > > tracer
> > > > > > > service by talking to ZooKeeper and observing its service
> > address
> > > > > > > advertisement, similar to how other servers register
themselves
> > in
> > > > > > > ZooKeeper. I'm not a docker network expert, but whatever
service
> > > > > address
> > > > > > > the tracer service is advertising there should be
routable from
> > > the
> > > > > > tablet
> > > > > > > servers.
> > > > > > >
> > > > > > > On Thu, Jul 28, 2022 at 7:31 PM kma <km...@phemi.com> wrote:
> > > > > > >
> > > > > > > > Thanks again Christopher,
> > > > > > > >
> > > > > > > > Our environment is a little different from usual.
> > > > > > > >
> > > > > > > > We have accumulo tracer running in it's own docker
container
> > and
> > > > the
> > > > > > > > other accumulo services (e.g. gc, master, tserver,
etc.) are
> > > also
> > > > > > > > running in different docker containers. We also have
kerberos
> > > > > enabled.
> > > > > > > >
> > > > > > > > The following is our trace related configurations
> > > > > > > >
> > > > > > > > default | trace.port.client ................. | 12234
> > > > > > > > default | trace.span.receivers .......... |
> > > > > > > > org.apache.accumulo.tracer.ZooTraceClient
> > > > > > > > default | trace.table ......................... | trace
> > > > > > > > default | trace.token.type ................ |
> > > > > > > >
org.apache.accumulo.core.client.security.tokens.PasswordToken
> > > > > > > > site | @override ...................... |
> > > > > > > >
org.apache.accumulo.core.client.security.tokens.KerberosToken
> > > > > > > > default | trace.user .......................... | root
> > > > > > > > site | @override ...................... |
> > > > > > > > accumulo-tra...@dev.phemi.com
> > > > > > > > default | trace.zookeeper.path ........ | /tracers
> > > > > > > >
> > > > > > > > And we can confirm that `trace on` and `trace off`
works well
> > in
> > > > the
> > > > > > > > accumulo tracer container where the tracer process is
running.
> > > > > > > >
> > > > > > > > However, `trace on` and `trace off` does not work in
any other
> > > > > > > > containers. This is probably why we don't see
compaction trace
> > > > > > messages
> > > > > > > > in the trace table. And it's likely because these
containers
> > > don't
> > > > > > know
> > > > > > > > where the tracer service is running ?
> > > > > > > >
> > > > > > > > Question: How does the other accumulo services know where
> > tracer
> > > > > > service
> > > > > > > > is running ?
> > > > > > > > Question: Is there a way to configure the tracer host where
> > the
> > > > > tracer
> > > > > > > > service is running ?
> > > > > > > >
> > > > > > > > Cheers!...
> > > > > > > > ...Keith
> > > > > > > >
> > > > > > > >
> > > > > > > > On 2022/07/22 14:58:07 Christopher wrote:
> > > > > > > > > I would double check your trace credentials in your
Accumulo
> > > > > > > > configuration
> > > > > > > > > file. And double check that the user credentials used to
> > write
> > > > to
> > > > > > the
> > > > > > > > trace
> > > > > > > > > table have permission to write to that table, and that
> > you're
> > > > > > running
> > > > > > > > the
> > > > > > > > > accumulo-tracer service.
> > > > > > > > >
> > > > > > > > > On Fri, Jul 22, 2022 at 10:45 AM kma <km...@phemi.com>
> > wrote:
> > > > > > > > >
> > > > > > > > > > Thank you Christopher,
> > > > > > > > > >
> > > > > > > > > > We can confirm that trace on and trace off are working
> > fine
> > > in
> > > > > > > > accumulo
> > > > > > > > > > shell. We can see those trace messages in the trace
table
> > > and
> > > > > > tracer
> > > > > > > > > > service is running.
> > > > > > > > > >
> > > > > > > > > > However, we are not seeing trace messages for MajC
/ MinC
> > > > > > > > > >
> > > > > > > > > > We do see these warnings in Monitor
> > > > > > > > > >
> > > > > > > > > > "Tracing spans are being dropped because there are
already
> > > > 5000
> > > > > > spans
> > > > > > > > > > queued for delivery. This does not affect performance,
> > > > security
> > > > > or
> > > > > > > > data
> > > > > > > > > > integrity, but distributed tracing information is being
> > > lost."
> > > > > > > > > >
> > > > > > > > > > Any suggestions ?
> > > > > > > > > >
> > > > > > > > > > Cheers!...
> > > > > > > > > > ...Keith
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 2022/07/12 19:42:07 Christopher wrote:
> > > > > > > > > > > By default, the tracer service that collects
traces and
> > > > writes
> > > > > > > > them to
> > > > > > > > > > > the trace table does not run. Generally, it is also
> > > > > > responsible for
> > > > > > > > > > > creating the trace table. If you have an empty trace
> > > table,
> > > > > > was it
> > > > > > > > > > > created by the tracer service, or did you create it
> > > > manually?
> > > > > > > > > > >
> > > > > > > > > > > Assuming tracing is working as intended and there
is a
> > > > tracer
> > > > > > > > service
> > > > > > > > > > > running, there's only a few operations that trace
> > > > > > automatically, and
> > > > > > > > > > > even they do so at a small probability by default
(like
> > > > > > > > compactions).
> > > > > > > > > > > If your system is just started and you're using
default
> > > > > > settings, it
> > > > > > > > > > > may not have had a chance to create any traces yet.
> > > > > > > > > > >
> > > > > > > > > > > Some operations can be traced by explicitly
turning on
> > > > > > tracing once
> > > > > > > > > > > everything is set up. You can test this by trying to
> > turn
> > > on
> > > > > > tracing
> > > > > > > > > > > in the shell, using the "trace on" command, and then
> > > doing a
> > > > > > scan,
> > > > > > > > > > > then doing "trace off". I believe this still works as
> > > > expected
> > > > > > in
> > > > > > > > > > > 2.0.1. This should trace client side operations
through
> > to
> > > > the
> > > > > > > > server,
> > > > > > > > > > > initiated from the shell. This is also possible to be
> > done
> > > > in
> > > > > > > > > > > user-code, but there is no public API for it.
> > > > > > > > > > >
> > > > > > > > > > > A lot of the behavior of tracing has had to change in
> > the
> > > > > > last few
> > > > > > > > > > > versions because of the dependencies we
previously used
> > > for
> > > > > > tracing
> > > > > > > > > > > are no longer supported. Everything should still work
> > like
> > > > it
> > > > > > did in
> > > > > > > > > > > previous versions, but it's possible 2.0.1 does have
> > > issues.
> > > > > For
> > > > > > > > 2.1,
> > > > > > > > > > > we're expecting to use OpenTelemetry, which
should make
> > > > > > tracing much
> > > > > > > > > > > easier and more reliable to configure. However, we do
> > not
> > > > > > provide a
> > > > > > > > > > > built-in OpenTelemetry sink in 2.1 to write tracing
> > > > > > information to a
> > > > > > > > > > > table in Accumulo, like we did in previous
versions for
> > > > > HTrace.
> > > > > > > > Such a
> > > > > > > > > > > thing could be added by the user or a 3rd party,
though.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jul 12, 2022 at 3:25 PM kma <km...@phemi.com>
> > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Team,
> > > > > > > > > > > >
> > > > > > > > > > > > In one of out latest deployment of Accumulo
2.0.1, the
> > > > trace
> > > > > > > > table is
> > > > > > > > > > > > empty.
> > > > > > > > > > > >
> > > > > > > > > > > > Question: is tracing automatically enabled by
default
> > ?
> > > If
> > > > > > not,
> > > > > > > > how to
> > > > > > > > > > > > enable it ?
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > ______________________________
> > > > > > > > > > > > Keith Ma
> > > > > > > > > > > > Software Developer
> > > > > > > > > > > > *PHEMI Systems*
> > > > > > > > > > > > Suite 600 – 777 Hornby Street
> > > > > > > > > > > > Vancouver, BC V6Z 1S4
> > > > > > > > > > > > 604-336-1119
> > > > > > > > > > > >
> > > > > > > > > > > > website <http://www.phemi.com/> twitter
> > > > > > > > > > > > <https://twitter.com/PHEMISystems>linkedin
> > > > > > > > > > > >
> > > > > > > > > > <
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > ______________________________
> > > > > > > > > > Keith Ma
> > > > > > > > > > Software Developer
> > > > > > > > > > *PHEMI Systems*
> > > > > > > > > > Suite 600 – 777 Hornby Street
> > > > > > > > > > Vancouver, BC V6Z 1S4
> > > > > > > > > > 604-336-1119
> > > > > > > > > >
> > > > > > > > > > website <http://www.phemi.com/> twitter
> > > > > > > > > > <https://twitter.com/PHEMISystems>linkedin
> > > > > > > > > > <
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > --
> > > > > > > > ______________________________
> > > > > > > > Keith Ma
> > > > > > > > Software Developer
> > > > > > > > *PHEMI Systems*
> > > > > > > > Suite 600 – 777 Hornby Street
> > > > > > > > Vancouver, BC V6Z 1S4
> > > > > > > > 604-336-1119
> > > > > > > >
> > > > > > > > website <http://www.phemi.com/> twitter
> > > > > > > > <https://twitter.com/PHEMISystems>linkedin
> > > > > > > > <
> > > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > --
> > > > > > ______________________________
> > > > > > Keith Ma
> > > > > > Software Developer
> > > > > > *PHEMI Systems*
> > > > > > Suite 600 – 777 Hornby Street
> > > > > > Vancouver, BC V6Z 1S4
> > > > > > 604-336-1119
> > > > > >
> > > > > > website <http://www.phemi.com/> twitter
> > > > > > <https://twitter.com/PHEMISystems>linkedin
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
______________________________
Keith Ma
Software Developer
*PHEMI Systems*
Suite 600 – 777 Hornby Street
Vancouver, BC V6Z 1S4
604-336-1119
website <http://www.phemi.com/> twitter
<https://twitter.com/PHEMISystems>linkedin
<http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1>
