steFaiz commented on PR #8191: URL: https://github.com/apache/paimon/pull/8191#issuecomment-4690080073
@JingsongLi Thanks for your advise! Do you mean failover of Spark/Flink jobs? If use java sdk, the write is two-phase commit. So no partial files will exist. (Sacrifising blob descriptors visibility). If do not use two-phase commit, partial files may exist. But after retry, emitted blobdescriptors are all valid (pointing to newly written files). And if use like OSS, only visiting blobs through BlobDescriptos, no scan, partial files could be omitted. Moreover, we could filter out partial files during read. Do you think this situation (simple replace OSS) not suitable for paimon? I'll close this pr and find another way to deal with it! Thanks again for your adivse! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
