On Fri, Feb 9, 2018 at 2:26 PM, Dong Zhou <dz...@phemi.com> wrote:
> Hi All,
>
> We were trying to write some data into an Accumulo table that contains
> roughly 3.7 Trillion entries using Bulk Import.
>
> Our code generates RFiles in distributed fashion, and most of them go into
> the table perfectly, excepting for 1 RFile.
>
> The bulk import failed with following error message. Please note that I have
> shortened the error message and also masked out some of the data.
>
>

This message is from an intermediate tserver.  It possible this
intermediate thread is running after the bulk import completed, more
on this below.

> 2018-02-05 21:47:07,244 [client.BulkImporter] ERROR: Encountered unknown
> exception in assignMapFiles.
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport
>         at
> org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
>         at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
>         at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_bulkImport(TabletClientService.java:594)
>         at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.bulkImport(TabletClientService.java:577)
>         at
> org.apache.accumulo.server.client.BulkImporter.assignMapFiles(BulkImporter.java:600)
>         at
> org.apache.accumulo.server.client.BulkImporter.access$400(BulkImporter.java:77)
>         at
> org.apache.accumulo.server.client.BulkImporter$AssignmentTask.run(BulkImporter.java:479)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>         at java.lang.Thread.run(Thread.java:745)
> 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
> files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport .  Will retry ...
> 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
> files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport .  Will retry ...
> 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
> files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport .  Will retry ...
> 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
> files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport .  Will retry ...
> 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
> files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport .  Will retry ...
> ...
> ...
> ...

Each bulk import has a transaction id in zookeeper.  When bulk import
finishes, it removes this transaction id from zookeeper (this is a
step in a bulk import FATE tx).  Writes are made to the metadata table
as part of bulk import. The metadata table has a constraint sanity
check that ensure the bulk import is active.  The exception below is
this check failing.

Below is the code that removes this Id from ZK.

https://github.com/apache/accumulo/blob/rel/1.8.1/server/master/src/main/java/org/apache/accumulo/master/tableOps/CompleteBulkImport.java#L42

Bulk import spawns work threads on intermediate tablet servers to
inspect rfiles indexes  to determine where the files should go.  These
intermediate tablet servers instruct other tablet servers to load
files.  The error message below is from one of these 2nd hop tservers.
Its possible that this spawned work thread is still running after the
bulk import has completed and this message could be ignored.  Did the
bulk import operation fail of succeed from the Accumulo client
perspective?   Its also possible that this error message is unrelated
to the reason the bulk import failed.  Was there one file in the fail
dir?

> 2018-02-05 21:47:07,494 [impl.Writer] ERROR: error sending update to
> <tserver ip address>:<tserver port>:
> ConstraintViolationException(violationSummaries:[TConstraintViolationSummary(constrainClass:org.apache.accumulo.server.constraints.MetadataConstraints,
> violationCode:8, violationDescription:Bulk load transaction no longer
> running, numberOfViolatingMutations:1)])
> 2018-02-05 21:47:07,494 [util.MetadataTableUtil] ERROR: null
> ConstraintViolationException(violationSummaries:[TConstraintViolationSummary(constrainClass:org.apache.accumulo.server.constraints.MetadataConstraints,
> violationCode:8, violationDescription:Bulk load transaction no longer
> running, numberOfViolatingMutations:1)])
>         at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$update_result$update_resultStandardScheme.read(TabletClientService.java:15928)
>         at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$update_result$update_resultStandardScheme.read(TabletClientService.java:15896)
>         at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$update_result.read(TabletClientService.java:15830)
>         at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
>         at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_update(TabletClientService.java:468)
>         at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.update(TabletClientService.java:451)
>         at
> org.apache.accumulo.core.client.impl.Writer.updateServer(Writer.java:72)
>         at
> org.apache.accumulo.core.client.impl.Writer.update(Writer.java:98)
>         at
> org.apache.accumulo.server.util.MetadataTableUtil.update(MetadataTableUtil.java:153)
>         at
> org.apache.accumulo.server.util.MetadataTableUtil.update(MetadataTableUtil.java:145)
>         at
> org.apache.accumulo.server.util.MetadataTableUtil.updateTabletDataFile(MetadataTableUtil.java:196)
>         at
> org.apache.accumulo.tserver.tablet.Tablet.updatePersistedTime(Tablet.java:2657)
>         at
> org.apache.accumulo.tserver.tablet.DatafileManager.importMapFiles(DatafileManager.java:268)
>         at
> org.apache.accumulo.tserver.tablet.Tablet.importMapFiles(Tablet.java:2376)
>         at
> org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.bulkImport(TabletServer.java:435)
>         at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
> org.apache.accumulo.core.trace.wrappers.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:46)
>         at
> org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:74)
>         at com.sun.proxy.$Proxy22.bulkImport(Unknown Source)
>         at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
> org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invokeMethod(TCredentialsUpdatingInvocationHandler.java:154)
>         at
> org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:58)
>         at com.sun.proxy.$Proxy22.bulkImport(Unknown Source)
>         at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$bulkImport.getResult(TabletClientService.java:2585)
>         at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$bulkImport.getResult(TabletClientService.java:2569)
>         at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>         at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>         at
> org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:102)
>         at
> org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
>         at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>         at java.lang.Thread.run(Thread.java:745)
> ...
> ...
> ...

I think this error message is from an intermediate tablet sever.

> 2018-02-05 21:47:07,526 [client.BulkImporter] ERROR: Encountered unknown
> exception in assignMapFiles.
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport
>         at
> org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
>         at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
>         at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_bulkImport(TabletClientService.java:594)
>         at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.bulkImport(TabletClientService.java:577)
>         at
> org.apache.accumulo.server.client.BulkImporter.assignMapFiles(BulkImporter.java:600)
>         at
> org.apache.accumulo.server.client.BulkImporter.access$400(BulkImporter.java:77)
>         at
> org.apache.accumulo.server.client.BulkImporter$AssignmentTask.run(BulkImporter.java:479)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>         at java.lang.Thread.run(Thread.java:745)
> ...
> ...
> ...
> 2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
> files to tablet o;checksum|****;checksum|**** because :
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport .  Will retry ...
> 2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
> files to tablet o;checksum|****;checksum|**** because :
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport .  Will retry ...
> 2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
> files to tablet o;checksum|****;checksum|**** because :
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport .  Will retry ...
> 2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
> files to tablet o;checksum|****;checksum|**** because :
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport .  Will retry ...
> 2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
> files to tablet o;checksum|****;checksum|**** because :
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport .  Will retry ...
>
>

Reply via email to