----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/75007/#review226593 -----------------------------------------------------------
Ship it! Ship It! - Madhan Neethiraj On June 25, 2024, 5:15 a.m., Pinal Shah wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/75007/ > ----------------------------------------------------------- > > (Updated June 25, 2024, 5:15 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > and Radhika Kundam. > > > Bugs: ATLAS-4866 > https://issues.apache.org/jira/browse/ATLAS-4866 > > > Repository: atlas > > > Description > ------- > > **Background:** > > Atlas uses HBase as its store for audit repository. > > After import, atlas store the audit entity with the import information along > with all the processed entitiy guids. > > **Issue: ** > > When large sized export zipped file is imported, import gives below error, > internally import gets succesful, but fail creating audit. > > *{"errorCode":"ATLAS-500-00-001","errorMessage":"org.janusgraph.core.JanusGraphException: > Could not commit transaction due to exception during > persistence","errorCause":"Could not commit transaction due to exception > during persistence"}* > > When size of the entity is greater than "hbase.client.keyvalue.maxsize" > property then audit entity creation fails with exception. > > *Caused by: > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 > action: org.apache.hadoop.hbase.DoNotRetryIOException: > Cell[\x00\x00\x00\x00\x00\x00\x00\x00\x01\x05\xCC\xBB/l:\x00\x06\x18r\xB0\xBE\xFDH\xA00a11ed186467-ve0214-halxg-cloudera-com\xB2\x00\x00\x00\x00\x00\x0D\xB6Y/1715730740890001/Put/vlen=23826488/seqid=0] > with size 23826581 exceeds limit of 10485760 bytes > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.checkCellSizeLimit(RSRpcServices.java:906) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:992) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:927) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:892) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2855) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45961) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:139) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) > : 1 time, servers with issues: ve0214.halxg.cloudera.com,22101,1715690875185 > at > org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50) > at > org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1228) > at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:434) > at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:422) > at > org.janusgraph.diskstorage.hbase2.HTable2_0.batch(HTable2_0.java:51) > > **Solution: ** > > Hence in this case, storing processed entities guids is skipped while > creating ExportImportAuditEntry when size of entity goes beyond the value (in > bytes) of below property > > atlas.hbase.client.keyvalue.maxsize > > > Diffs > ----- > > > repository/src/main/java/org/apache/atlas/repository/impexp/ExportImportAuditService.java > 3afa17301 > > > Diff: https://reviews.apache.org/r/75007/diff/2/ > > > Testing > ------- > > manually verified through below api, processed entity guids is not stored > > /api/atlas/admin/expimp/audit?userName=admin&operation=IMPORT > > Precommit: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/1649/console > > > Thanks, > > Pinal Shah > >
