Nice,
Turn on debug really help.
2022-07-29 12:12:14,235 [tracer.ZooTraceClient] DEBUG: Scanning trace
hosts in zookeeper: /tracers
2022-07-29 12:12:14,240 [tracer.ZooTraceClient] DEBUG: Trace hosts:
[0.0.0.0:12234, 0.0.0.0:12234]
2022-07-29 12:12:14,240 [tracer.ZooTraceClient] DEBUG: Successfully
initialized tracer hosts from ZooKeeper
[root@accumulo_gc /]# curl 0.0.0.0:12234
curl: (7) Failed connect to 0.0.0.0:12234; Connection refused
[root@accumulo_gc /]# curl accumulo_tracer:12234
curl: (52) Empty reply from server
Is there a way to set Trace hosts to accumulo_tracer:12234
Cheers!...
...Keith
On 2022/07/28 23:40:18 Christopher wrote:
> Do all servers have the same configuration?
>
> I would investigate the tablet server debug logs to determine if it's
> having trouble setting up tracing. It should be able to locate the tracer
> service by talking to ZooKeeper and observing its service address
> advertisement, similar to how other servers register themselves in
> ZooKeeper. I'm not a docker network expert, but whatever service address
> the tracer service is advertising there should be routable from the
tablet
> servers.
>
> On Thu, Jul 28, 2022 at 7:31 PM kma <km...@phemi.com> wrote:
>
> > Thanks again Christopher,
> >
> > Our environment is a little different from usual.
> >
> > We have accumulo tracer running in it's own docker container and the
> > other accumulo services (e.g. gc, master, tserver, etc.) are also
> > running in different docker containers. We also have kerberos enabled.
> >
> > The following is our trace related configurations
> >
> > default | trace.port.client ................. | 12234
> > default | trace.span.receivers .......... |
> > org.apache.accumulo.tracer.ZooTraceClient
> > default | trace.table ......................... | trace
> > default | trace.token.type ................ |
> > org.apache.accumulo.core.client.security.tokens.PasswordToken
> > site | @override ...................... |
> > org.apache.accumulo.core.client.security.tokens.KerberosToken
> > default | trace.user .......................... | root
> > site | @override ...................... |
> > accumulo-tra...@dev.phemi.com
> > default | trace.zookeeper.path ........ | /tracers
> >
> > And we can confirm that `trace on` and `trace off` works well in the
> > accumulo tracer container where the tracer process is running.
> >
> > However, `trace on` and `trace off` does not work in any other
> > containers. This is probably why we don't see compaction trace messages
> > in the trace table. And it's likely because these containers don't know
> > where the tracer service is running ?
> >
> > Question: How does the other accumulo services know where tracer
service
> > is running ?
> > Question: Is there a way to configure the tracer host where the tracer
> > service is running ?
> >
> > Cheers!...
> > ...Keith
> >
> >
> > On 2022/07/22 14:58:07 Christopher wrote:
> > > I would double check your trace credentials in your Accumulo
> > configuration
> > > file. And double check that the user credentials used to write to the
> > trace
> > > table have permission to write to that table, and that you're running
> > the
> > > accumulo-tracer service.
> > >
> > > On Fri, Jul 22, 2022 at 10:45 AM kma <km...@phemi.com> wrote:
> > >
> > > > Thank you Christopher,
> > > >
> > > > We can confirm that trace on and trace off are working fine in
> > accumulo
> > > > shell. We can see those trace messages in the trace table and
tracer
> > > > service is running.
> > > >
> > > > However, we are not seeing trace messages for MajC / MinC
> > > >
> > > > We do see these warnings in Monitor
> > > >
> > > > "Tracing spans are being dropped because there are already 5000
spans
> > > > queued for delivery. This does not affect performance, security or
> > data
> > > > integrity, but distributed tracing information is being lost."
> > > >
> > > > Any suggestions ?
> > > >
> > > > Cheers!...
> > > > ...Keith
> > > >
> > > >
> > > > On 2022/07/12 19:42:07 Christopher wrote:
> > > > > By default, the tracer service that collects traces and writes
> > them to
> > > > > the trace table does not run. Generally, it is also
responsible for
> > > > > creating the trace table. If you have an empty trace table,
was it
> > > > > created by the tracer service, or did you create it manually?
> > > > >
> > > > > Assuming tracing is working as intended and there is a tracer
> > service
> > > > > running, there's only a few operations that trace
automatically, and
> > > > > even they do so at a small probability by default (like
> > compactions).
> > > > > If your system is just started and you're using default
settings, it
> > > > > may not have had a chance to create any traces yet.
> > > > >
> > > > > Some operations can be traced by explicitly turning on
tracing once
> > > > > everything is set up. You can test this by trying to turn on
tracing
> > > > > in the shell, using the "trace on" command, and then doing a
scan,
> > > > > then doing "trace off". I believe this still works as expected in
> > > > > 2.0.1. This should trace client side operations through to the
> > server,
> > > > > initiated from the shell. This is also possible to be done in
> > > > > user-code, but there is no public API for it.
> > > > >
> > > > > A lot of the behavior of tracing has had to change in the
last few
> > > > > versions because of the dependencies we previously used for
tracing
> > > > > are no longer supported. Everything should still work like it
did in
> > > > > previous versions, but it's possible 2.0.1 does have issues. For
> > 2.1,
> > > > > we're expecting to use OpenTelemetry, which should make
tracing much
> > > > > easier and more reliable to configure. However, we do not
provide a
> > > > > built-in OpenTelemetry sink in 2.1 to write tracing
information to a
> > > > > table in Accumulo, like we did in previous versions for HTrace.
> > Such a
> > > > > thing could be added by the user or a 3rd party, though.
> > > > >
> > > > > On Tue, Jul 12, 2022 at 3:25 PM kma <km...@phemi.com> wrote:
> > > > > >
> > > > > > Hi Team,
> > > > > >
> > > > > > In one of out latest deployment of Accumulo 2.0.1, the trace
> > table is
> > > > > > empty.
> > > > > >
> > > > > > Question: is tracing automatically enabled by default ? If not,
> > how to
> > > > > > enable it ?
> > > > > >
> > > > > > --
> > > > > > ______________________________
> > > > > > Keith Ma
> > > > > > Software Developer
> > > > > > *PHEMI Systems*
> > > > > > Suite 600 – 777 Hornby Street
> > > > > > Vancouver, BC V6Z 1S4
> > > > > > 604-336-1119
> > > > > >
> > > > > > website <http://www.phemi.com/> twitter
> > > > > > <https://twitter.com/PHEMISystems>linkedin
> > > > > >
> > > > <
> > > >
> >
> >
http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1
> > > > >
> > > > > >
> > > > >
> > > > --
> > > > ______________________________
> > > > Keith Ma
> > > > Software Developer
> > > > *PHEMI Systems*
> > > > Suite 600 – 777 Hornby Street
> > > > Vancouver, BC V6Z 1S4
> > > > 604-336-1119
> > > >
> > > > website <http://www.phemi.com/> twitter
> > > > <https://twitter.com/PHEMISystems>linkedin
> > > > <
> > > >
> >
> >
http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1
> > > > >
> > > >
> > > >
> > >
> > --
> > ______________________________
> > Keith Ma
> > Software Developer
> > *PHEMI Systems*
> > Suite 600 – 777 Hornby Street
> > Vancouver, BC V6Z 1S4
> > 604-336-1119
> >
> > website <http://www.phemi.com/> twitter
> > <https://twitter.com/PHEMISystems>linkedin
> > <
> >
http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1
> > >
> >
> >
>
--
______________________________
Keith Ma
Software Developer
*PHEMI Systems*
Suite 600 – 777 Hornby Street
Vancouver, BC V6Z 1S4
604-336-1119
website <http://www.phemi.com/> twitter
<https://twitter.com/PHEMISystems>linkedin
<http://www.linkedin.com/company/3561810?trk=tyah&trkInfo=tarId:1403279580554,tas:phemi%20hea,idx:1-1-1>
