hycdong commented on issue #1083:
URL:
https://github.com/apache/incubator-pegasus/issues/1083#issuecomment-1197545328
There are two cases which will cause bulk load ingestion data inconsistence:
- some partitions meet unrecoverable ingestion error during ingestion and
some not
- simple network error or replica 2pc are NOT unrecoverable error, only
the ingested files can not be recognized by rocksdb is unrecoverable
- force cancel during ingestion - this is only triggered by user not by
system itself
There are currently no solution to handle such situation automatically by
system, because pegasus doesn't support transaction through different
partitions. For example, table has 8 partitions. Partition 0 ingest succeed,
but partition 1 receive wrong-format sst files which is a unrecoverable error,
it can not ingest those files, bulk load failed, and partition 0 won't reset
those data. Ingestion is just like batch write, different partitions won't
affect others' data, they are just different partitions. If user use our client
batch write interface, it also can not gurantee that data wrote into different
partitions should always be consistent.
The only solution is to retry bulk load after user fix broken files, which
is triggered by user manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
