[
https://issues.apache.org/jira/browse/ACCUMULO-4506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634273#comment-15634273
]
Josh Elser commented on ACCUMULO-4506:
--------------------------------------
Good finding, [~adamjshook]! Sorry that the log message doesn't report the
TabletServer which should be doing the work.
It's possible that the Master got confused as it's maintaining some state in
memory and using ZK to have a TServer do the replication task.
Check if the following record is still in ZK:
{{/replication/workqueue/ae4b03ec-159b-44e8-9a88-ccf7fa849c19|peer_instance|...}}
(something something -- that should be enough of a prefix of the znode. The
beginning part of the znode will be the WAL's filename).
Has the master been running the whole time? It's possible that the Master's
internal state ({{UnorderedWorkAssigner}}'s {{queuedWork}} member) is screwed
up and a restart will kick it back into shape.
Looking at the code, the following appears wrong to me:
{code}
@Override
protected void cleanupFinishedWork() {
final Iterator<String> work = queuedWork.iterator();
final String instanceId = conn.getInstance().getInstanceID();
while (work.hasNext()) {
String filename = work.next();
// Null equates to the work was finished
if (null == zooCache.get(ZooUtil.getRoot(instanceId) +
ReplicationConstants.ZOO_WORK_QUEUE + "/" + filename)) {
work.remove();
}
}
}
{code}
I would think that {{filename}} should really be
{{DistributedWorkQueueWorkAssignerHelper.getQueueKey(walFileName,
replicationTarget)}}. Similarly, {{getQueuedWork()}} should use the
{{#getQueueKey(..)}} method and it isn't. My hunch is that if the above *is*
checking ZK for the wrong node, we're incorrectly keeping the memory in sync
with ZK. Might you be able to poke at this further, Adam?
> Some in-progress files for replication never replicate
> -------------------------------------------------------
>
> Key: ACCUMULO-4506
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4506
> Project: Accumulo
> Issue Type: Bug
> Components: replication
> Affects Versions: 1.7.2
> Reporter: Adam J Shook
>
> We're seeing an issue with replication where two files have been in-progress
> for a long time and based on the logs are not going to be replicated. The
> metadata from the {{accumulo.replication}} table looks a little funky, with a
> very large {{begin}} value.
> *Logs*
> {noformat}
> 2016-11-02 19:52:50,900 [replication.DistributedWorkQueueWorkAssigner] DEBUG:
> Not queueing work for
> hdfs://host:9000/accumulo/wal/host+31032/9f038f64-4252-44a0-bfd0-99d4a316b397
> to Remote Name: peer_instance Remote identifier: 5h Source Table ID: k
> because [begin: 9223372036854775807 end: 0 infiniteEnd: true closed: true
> createdTime: 1477314365827] doesn't need replication
> 2016-11-02 19:53:08,900 [replication.DistributedWorkQueueWorkAssigner] DEBUG:
> Not queueing work for
> hdfs://host:9000/accumulo/wal/host+31368/ae4b03ec-159b-44e8-9a88-ccf7fa849c19
> to Remote Name: peer_instance Remote identifier: 5i Source Table ID: l
> because [begin: 9223372036854775807 end: 0 infiniteEnd: true closed: true
> createdTime: 1477052816174] doesn't need replication
> {noformat}
> *Replication table*
> {noformat}
> scan -r
> hdfs://host:9000/accumulo/wal/host+31032/9f038f64-4252-44a0-bfd0-99d4a316b397
> -t accumulo.replication
> hdfs://host:9000/accumulo/wal/host+31032/9f038f64-4252-44a0-bfd0-99d4a316b397
> repl:j [] [begin: 0 end: 0 infiniteEnd: true closed: true createdTime:
> 1477314369633]
> hdfs://host:9000/accumulo/wal/host+31032/9f038f64-4252-44a0-bfd0-99d4a316b397
> repl:k [] [begin: 9223372036854775807 end: 0 infiniteEnd: true closed:
> true createdTime: 1477314365827]
> hdfs://host:9000/accumulo/wal/host+31032/9f038f64-4252-44a0-bfd0-99d4a316b397
> repl:l [] [begin: 9223372036854775807 end: 0 infiniteEnd: true closed:
> true createdTime: 1477314365707]
> hdfs://host:9000/accumulo/wal/host+31032/9f038f64-4252-44a0-bfd0-99d4a316b397
> work:\x01\x00\x00\x00\x17peer_instance\x01\x00\x00\x00\x025g\x01\x00\x00\x00\x01j
> [] [begin: 0 end: 0 infiniteEnd: true closed: true createdTime:
> 1477314369633]
> hdfs://host:9000/accumulo/wal/host+31032/9f038f64-4252-44a0-bfd0-99d4a316b397
> work:\x01\x00\x00\x00\x17peer_instance\x01\x00\x00\x00\x025h\x01\x00\x00\x00\x01k
> [] [begin: 9223372036854775807 end: 0 infiniteEnd: true closed: true
> createdTime: 1477314365827]
> hdfs://host:9000/accumulo/wal/host+31032/9f038f64-4252-44a0-bfd0-99d4a316b397
> work:\x01\x00\x00\x00\x17peer_instance\x01\x00\x00\x00\x025i\x01\x00\x00\x00\x01l
> [] [begin: 9223372036854775807 end: 0 infiniteEnd: true closed: true
> createdTime: 1477314365707]
> scan -r
> hdfs://host:9000/accumulo/wal/host+31368/ae4b03ec-159b-44e8-9a88-ccf7fa849c19
> -t accumulo.replication
> hdfs://host:9000/accumulo/wal/host+31368/ae4b03ec-159b-44e8-9a88-ccf7fa849c19
> repl:j [] [begin: 9223372036854775807 end: 0 infiniteEnd: true closed:
> true createdTime: 1477052819752]
> hdfs://host:9000/accumulo/wal/host+31368/ae4b03ec-159b-44e8-9a88-ccf7fa849c19
> repl:k [] [begin: 0 end: 0 infiniteEnd: true closed: true createdTime:
> 1477052816238]
> hdfs://host:9000/accumulo/wal/host+31368/ae4b03ec-159b-44e8-9a88-ccf7fa849c19
> repl:l [] [begin: 9223372036854775807 end: 0 infiniteEnd: true closed:
> true createdTime: 1477052816174]
> hdfs://host:9000/accumulo/wal/host+31368/ae4b03ec-159b-44e8-9a88-ccf7fa849c19
> work:\x01\x00\x00\x00\x17peer_instance\x01\x00\x00\x00\x025g\x01\x00\x00\x00\x01j
> [] [begin: 9223372036854775807 end: 0 infiniteEnd: true closed: true
> createdTime: 1477052819752]
> hdfs://host:9000/accumulo/wal/host+31368/ae4b03ec-159b-44e8-9a88-ccf7fa849c19
> work:\x01\x00\x00\x00\x17peer_instance\x01\x00\x00\x00\x025h\x01\x00\x00\x00\x01k
> [] [begin: 0 end: 0 infiniteEnd: true closed: true createdTime:
> 1477052816238]
> hdfs://host:9000/accumulo/wal/host+31368/ae4b03ec-159b-44e8-9a88-ccf7fa849c19
> work:\x01\x00\x00\x00\x17peer_instance\x01\x00\x00\x00\x025i\x01\x00\x00\x00\x01l
> [] [begin: 9223372036854775807 end: 0 infiniteEnd: true closed: true
> createdTime: 1477052816174]
> {noformat}
> *HDFS*
> {noformat}
> hdfs dfs -ls
> hdfs://host:9000/accumulo/wal/host+31032/9f038f64-4252-44a0-bfd0-99d4a316b397
> hdfs://host:9000/accumulo/wal/host+31368/ae4b03ec-159b-44e8-9a88-ccf7fa849c19
> -rwxr-xr-x 3 ubuntu supergroup 1117650900 2016-10-24 13:09
> hdfs://host:9000/accumulo/wal/host+31032/9f038f64-4252-44a0-bfd0-99d4a316b397
> -rwxr-xr-x 3 ubuntu supergroup 1171968390 2016-10-21 12:31
> hdfs://host:9000/accumulo/wal/host+31368/ae4b03ec-159b-44e8-9a88-ccf7fa849c19
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
