The bulk import operation did finish, but it was marked as failed.
The failed to import rfile was moved into the failure directory.

We tried manually to import this rfile using importdirectory, and then the
exact same outcome occurred and error log appeared in the tserver log file.


On Fri, Feb 9, 2018 at 1:46 PM Keith Turner <ke...@deenlo.com> wrote:

> On Fri, Feb 9, 2018 at 2:26 PM, Dong Zhou <dz...@phemi.com> wrote:
> > Hi All,
> >
> > We were trying to write some data into an Accumulo table that contains
> > roughly 3.7 Trillion entries using Bulk Import.
> >
> > Our code generates RFiles in distributed fashion, and most of them go
> into
> > the table perfectly, excepting for 1 RFile.
> >
> > The bulk import failed with following error message. Please note that I
> have
> > shortened the error message and also masked out some of the data.
> >
> >
>
> This message is from an intermediate tserver.  It possible this
> intermediate thread is running after the bulk import completed, more
> on this below.
>
> > 2018-02-05 21:47:07,244 [client.BulkImporter] ERROR: Encountered unknown
> > exception in assignMapFiles.
> > org.apache.thrift.TApplicationException: Internal error processing
> > bulkImport
> >         at
> >
> org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
> >         at
> > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
> >         at
> >
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_bulkImport(TabletClientService.java:594)
> >         at
> >
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.bulkImport(TabletClientService.java:577)
> >         at
> >
> org.apache.accumulo.server.client.BulkImporter.assignMapFiles(BulkImporter.java:600)
> >         at
> >
> org.apache.accumulo.server.client.BulkImporter.access$400(BulkImporter.java:77)
> >         at
> >
> org.apache.accumulo.server.client.BulkImporter$AssignmentTask.run(BulkImporter.java:479)
> >         at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >         at
> >
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> >         at java.lang.Thread.run(Thread.java:745)
> > 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1
> map
> > files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
> > org.apache.thrift.TApplicationException: Internal error processing
> > bulkImport .  Will retry ...
> > 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1
> map
> > files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
> > org.apache.thrift.TApplicationException: Internal error processing
> > bulkImport .  Will retry ...
> > 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1
> map
> > files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
> > org.apache.thrift.TApplicationException: Internal error processing
> > bulkImport .  Will retry ...
> > 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1
> map
> > files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
> > org.apache.thrift.TApplicationException: Internal error processing
> > bulkImport .  Will retry ...
> > 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1
> map
> > files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
> > org.apache.thrift.TApplicationException: Internal error processing
> > bulkImport .  Will retry ...
> > ...
> > ...
> > ...
>
> Each bulk import has a transaction id in zookeeper.  When bulk import
> finishes, it removes this transaction id from zookeeper (this is a
> step in a bulk import FATE tx).  Writes are made to the metadata table
> as part of bulk import. The metadata table has a constraint sanity
> check that ensure the bulk import is active.  The exception below is
> this check failing.
>
> Below is the code that removes this Id from ZK.
>
>
> https://github.com/apache/accumulo/blob/rel/1.8.1/server/master/src/main/java/org/apache/accumulo/master/tableOps/CompleteBulkImport.java#L42
>
> Bulk import spawns work threads on intermediate tablet servers to
> inspect rfiles indexes  to determine where the files should go.  These
> intermediate tablet servers instruct other tablet servers to load
> files.  The error message below is from one of these 2nd hop tservers.
> Its possible that this spawned work thread is still running after the
> bulk import has completed and this message could be ignored.  Did the
> bulk import operation fail of succeed from the Accumulo client
> perspective?   Its also possible that this error message is unrelated
> to the reason the bulk import failed.  Was there one file in the fail
> dir?
>
> > 2018-02-05 21:47:07,494 [impl.Writer] ERROR: error sending update to
> > <tserver ip address>:<tserver port>:
> >
> ConstraintViolationException(violationSummaries:[TConstraintViolationSummary(constrainClass:org.apache.accumulo.server.constraints.MetadataConstraints,
> > violationCode:8, violationDescription:Bulk load transaction no longer
> > running, numberOfViolatingMutations:1)])
> > 2018-02-05 21:47:07,494 [util.MetadataTableUtil] ERROR: null
> >
> ConstraintViolationException(violationSummaries:[TConstraintViolationSummary(constrainClass:org.apache.accumulo.server.constraints.MetadataConstraints,
> > violationCode:8, violationDescription:Bulk load transaction no longer
> > running, numberOfViolatingMutations:1)])
> >         at
> >
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$update_result$update_resultStandardScheme.read(TabletClientService.java:15928)
> >         at
> >
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$update_result$update_resultStandardScheme.read(TabletClientService.java:15896)
> >         at
> >
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$update_result.read(TabletClientService.java:15830)
> >         at
> > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> >         at
> >
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_update(TabletClientService.java:468)
> >         at
> >
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.update(TabletClientService.java:451)
> >         at
> > org.apache.accumulo.core.client.impl.Writer.updateServer(Writer.java:72)
> >         at
> > org.apache.accumulo.core.client.impl.Writer.update(Writer.java:98)
> >         at
> >
> org.apache.accumulo.server.util.MetadataTableUtil.update(MetadataTableUtil.java:153)
> >         at
> >
> org.apache.accumulo.server.util.MetadataTableUtil.update(MetadataTableUtil.java:145)
> >         at
> >
> org.apache.accumulo.server.util.MetadataTableUtil.updateTabletDataFile(MetadataTableUtil.java:196)
> >         at
> >
> org.apache.accumulo.tserver.tablet.Tablet.updatePersistedTime(Tablet.java:2657)
> >         at
> >
> org.apache.accumulo.tserver.tablet.DatafileManager.importMapFiles(DatafileManager.java:268)
> >         at
> >
> org.apache.accumulo.tserver.tablet.Tablet.importMapFiles(Tablet.java:2376)
> >         at
> >
> org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.bulkImport(TabletServer.java:435)
> >         at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
> >         at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:498)
> >         at
> >
> org.apache.accumulo.core.trace.wrappers.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:46)
> >         at
> > org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:74)
> >         at com.sun.proxy.$Proxy22.bulkImport(Unknown Source)
> >         at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
> >         at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:498)
> >         at
> >
> org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invokeMethod(TCredentialsUpdatingInvocationHandler.java:154)
> >         at
> >
> org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:58)
> >         at com.sun.proxy.$Proxy22.bulkImport(Unknown Source)
> >         at
> >
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$bulkImport.getResult(TabletClientService.java:2585)
> >         at
> >
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$bulkImport.getResult(TabletClientService.java:2569)
> >         at
> > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> >         at
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> >         at
> >
> org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:102)
> >         at
> >
> org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
> >         at
> >
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >         at
> >
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> >         at java.lang.Thread.run(Thread.java:745)
> > ...
> > ...
> > ...
>
> I think this error message is from an intermediate tablet sever.
>
> > 2018-02-05 21:47:07,526 [client.BulkImporter] ERROR: Encountered unknown
> > exception in assignMapFiles.
> > org.apache.thrift.TApplicationException: Internal error processing
> > bulkImport
> >         at
> >
> org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
> >         at
> > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
> >         at
> >
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_bulkImport(TabletClientService.java:594)
> >         at
> >
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.bulkImport(TabletClientService.java:577)
> >         at
> >
> org.apache.accumulo.server.client.BulkImporter.assignMapFiles(BulkImporter.java:600)
> >         at
> >
> org.apache.accumulo.server.client.BulkImporter.access$400(BulkImporter.java:77)
> >         at
> >
> org.apache.accumulo.server.client.BulkImporter$AssignmentTask.run(BulkImporter.java:479)
> >         at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >         at
> >
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> >         at java.lang.Thread.run(Thread.java:745)
> > ...
> > ...
> > ...
> > 2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1
> map
> > files to tablet o;checksum|****;checksum|**** because :
> > org.apache.thrift.TApplicationException: Internal error processing
> > bulkImport .  Will retry ...
> > 2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1
> map
> > files to tablet o;checksum|****;checksum|**** because :
> > org.apache.thrift.TApplicationException: Internal error processing
> > bulkImport .  Will retry ...
> > 2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1
> map
> > files to tablet o;checksum|****;checksum|**** because :
> > org.apache.thrift.TApplicationException: Internal error processing
> > bulkImport .  Will retry ...
> > 2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1
> map
> > files to tablet o;checksum|****;checksum|**** because :
> > org.apache.thrift.TApplicationException: Internal error processing
> > bulkImport .  Will retry ...
> > 2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1
> map
> > files to tablet o;checksum|****;checksum|**** because :
> > org.apache.thrift.TApplicationException: Internal error processing
> > bulkImport .  Will retry ...
> >
> >
>

Reply via email to