Sean Busbey created ACCUMULO-2014:
-------------------------------------
Summary: Feedback loop on Tracer table given small in-memory map
on server
Key: ACCUMULO-2014
URL: https://issues.apache.org/jira/browse/ACCUMULO-2014
Project: Accumulo
Issue Type: Bug
Components: trace, tserver
Affects Versions: 1.4.3
Reporter: Sean Busbey
Priority: Minor
Under the right conditions the tracer table can get into a feedback loop where
messages about flushes to a tracer table tablet keep arriving and triggering
subsequent flush operations.
This was observed on a 1.4.3 cluster initially configured for 256MB in-memory
maps.
This manifests as a high rate of ingest to the tracer table, even in the
absence of client activity or any client created tables. i.e. on a 3 worker
node cluster with no external activity 500 entries/second into the tracer table.
Viewing the tserver logs:
{noformat}
tserver.log (less AUDIT):
$ tail -50 tserver-3-example.com.log | grep -v AUDIT
2013-12-10 19:08:22,433 INFO
org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent
1<;b99f as alias 5
2013-12-10 19:08:22,627 INFO
org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent
!0<;~ as alias 2
2013-12-10 19:08:23,431 INFO
org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent
1<;b99f as alias 5
2013-12-10 19:08:23,735 INFO
org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent
!0<;~ as alias 2
2013-12-10 19:08:24,431 INFO
org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent
1<;b99f as alias 5
2013-12-10 19:08:24,875 INFO
org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent
!0<;~ as alias 2
2013-12-10 19:08:26,431 INFO
org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent
1<;b99f as alias 5
2013-12-10 19:08:26,486 INFO
org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent
!0<;~ as alias 2
tserver.log (full)
2013-12-10 19:07:44,479 AUDIT org.apache.accumulo.server.security.Auditor:
Using credentials root: authenticated
2013-12-10 19:07:44,480 AUDIT org.apache.accumulo.server.security.Auditor:
Using credentials !SYSTEM: checked permission WRITE on table 1 for root
2013-12-10 19:07:44,561 AUDIT org.apache.accumulo.server.security.Auditor:
Using credentials !SYSTEM: checked permission WRITE on table !0 for !SYSTEM
2013-12-10 19:07:45,379 AUDIT org.apache.accumulo.server.security.Auditor:
Using credentials !SYSTEM: checked permission READ on table !0 for !SYSTEM
2013-12-10 19:07:45,379 AUDIT org.apache.accumulo.server.security.Auditor:
Using credentials !SYSTEM: got authorizations for !SYSTEM
2013-12-10 19:07:45,479 AUDIT org.apache.accumulo.server.security.Auditor:
Using credentials root: authenticated
2013-12-10 19:07:45,480 AUDIT org.apache.accumulo.server.security.Auditor:
Using credentials !SYSTEM: checked permission WRITE on table 1 for root
2013-12-10 19:07:45,481 INFO
org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent
1<;b99f as alias 5
2013-12-10 19:07:45,554 AUDIT org.apache.accumulo.server.security.Auditor:
Using credentials !SYSTEM: checked permission WRITE on table !0 for !SYSTEM
2013-12-10 19:07:45,555 INFO
org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent
!0<;~ as alias 2
2013-12-10 19:07:45,731 AUDIT org.apache.accumulo.server.security.Auditor:
Using credentials !SYSTEM: checked permission WRITE on table !0 for !SYSTEM
2013-12-10 19:07:46,479 AUDIT org.apache.accumulo.server.security.Auditor:
Using credentials root: authenticated
2013-12-10 19:07:46,481 AUDIT org.apache.accumulo.server.security.Auditor:
Using credentials !SYSTEM: checked permission WRITE on table 1 for root
2013-12-10 19:07:46,482 INFO
org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent
1<;b99f as alias 5
2013-12-10 19:07:46,574 AUDIT org.apache.accumulo.server.security.Auditor:
Using credentials !SYSTEM: checked permission WRITE on table !0 for !SYSTEM
2013-12-10 19:07:46,575 INFO
org.apache.accumulo.server.tabletserver.TabletServer: Adding 1 logs for extent
!0<;~ as alias 2
2013-12-10 19:07:46,743 AUDIT org.apache.accumulo.server.security.Auditor:
Using credentials !SYSTEM: checked permission WRITE on table !0 for !SYSTEM
{noformat}
note that this example relies on the default configuration, using "root" as the
user for the tracer role. If your tracer user is set to something else, that is
the user that would show up in the credential checks above.
Current work around is to increase the memory map size.
We should attempt to detect this situation and at least issue a clear log
message. It would be nicer if we could detect it and turn off tracing messages
related to the tracing table.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
