Hi All,

We were trying to write some data into an Accumulo table that contains
roughly 3.7 Trillion entries using Bulk Import.

Our code generates RFiles in distributed fashion, and most of them go into
the table perfectly, excepting for 1 RFile.

The bulk import failed with following error message. Please note that I
have shortened the error message and also masked out some of the data.

2018-02-05 21:47:07,244 [client.BulkImporter] ERROR: Encountered unknown
exception in assignMapFiles.
org.apache.thrift.TApplicationException: Internal error processing
bulkImport
        at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
        at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_bulkImport(TabletClientService.java:594)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.bulkImport(TabletClientService.java:577)
        at
org.apache.accumulo.server.client.BulkImporter.assignMapFiles(BulkImporter.java:600)
        at
org.apache.accumulo.server.client.BulkImporter.access$400(BulkImporter.java:77)
        at
org.apache.accumulo.server.client.BulkImporter$AssignmentTask.run(BulkImporter.java:479)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:745)
2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
...
...
...
2018-02-05 21:47:07,494 [impl.Writer] ERROR: error sending update to
<tserver ip address>:<tserver port>:
ConstraintViolationException(violationSummaries:[TConstraintViolationSummary(constrainClass:org.apache.accumulo.server.constraints.MetadataConstraints,
violationCode:8, violationDescription:Bulk load transaction no longer
running, numberOfViolatingMutations:1)])
2018-02-05 21:47:07,494 [util.MetadataTableUtil] ERROR: null
ConstraintViolationException(violationSummaries:[TConstraintViolationSummary(constrainClass:org.apache.accumulo.server.constraints.MetadataConstraints,
violationCode:8, violationDescription:Bulk load transaction no longer
running, numberOfViolatingMutations:1)])
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$update_result$update_resultStandardScheme.read(TabletClientService.java:15928)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$update_result$update_resultStandardScheme.read(TabletClientService.java:15896)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$update_result.read(TabletClientService.java:15830)
        at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_update(TabletClientService.java:468)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.update(TabletClientService.java:451)
        at
org.apache.accumulo.core.client.impl.Writer.updateServer(Writer.java:72)
        at
org.apache.accumulo.core.client.impl.Writer.update(Writer.java:98)
        at
org.apache.accumulo.server.util.MetadataTableUtil.update(MetadataTableUtil.java:153)
        at
org.apache.accumulo.server.util.MetadataTableUtil.update(MetadataTableUtil.java:145)
        at
org.apache.accumulo.server.util.MetadataTableUtil.updateTabletDataFile(MetadataTableUtil.java:196)
        at
org.apache.accumulo.tserver.tablet.Tablet.updatePersistedTime(Tablet.java:2657)
        at
org.apache.accumulo.tserver.tablet.DatafileManager.importMapFiles(DatafileManager.java:268)
        at
org.apache.accumulo.tserver.tablet.Tablet.importMapFiles(Tablet.java:2376)
        at
org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.bulkImport(TabletServer.java:435)
        at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.accumulo.core.trace.wrappers.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:46)
        at
org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:74)
        at com.sun.proxy.$Proxy22.bulkImport(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invokeMethod(TCredentialsUpdatingInvocationHandler.java:154)
        at
org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:58)
        at com.sun.proxy.$Proxy22.bulkImport(Unknown Source)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$bulkImport.getResult(TabletClientService.java:2585)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$bulkImport.getResult(TabletClientService.java:2569)
        at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at
org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:102)
        at
org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:745)
...
...
...
2018-02-05 21:47:07,526 [client.BulkImporter] ERROR: Encountered unknown
exception in assignMapFiles.
org.apache.thrift.TApplicationException: Internal error processing
bulkImport
        at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
        at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_bulkImport(TabletClientService.java:594)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.bulkImport(TabletClientService.java:577)
        at
org.apache.accumulo.server.client.BulkImporter.assignMapFiles(BulkImporter.java:600)
        at
org.apache.accumulo.server.client.BulkImporter.access$400(BulkImporter.java:77)
        at
org.apache.accumulo.server.client.BulkImporter$AssignmentTask.run(BulkImporter.java:479)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:745)
...
...
...
2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;checksum|****;checksum|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;checksum|****;checksum|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;checksum|****;checksum|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;checksum|****;checksum|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;checksum|****;checksum|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...

Reply via email to