xintongsong commented on pull request #15599: URL: https://github.com/apache/flink/pull/15599#issuecomment-862042096
Thanks for the info, @galenwarren. I guess I'll wait for the remaining being done. Concerning the testability, ideally yes. It would be nice to have test cases covering behaviors of both general interfaces (FileSystem, RecoverableWriter, etc.) and the internal GCS specific components, and the standard test suites are supposed to make the former easier. However, these standard test suites requires corresponding external storage services to be available. E.g., HDFS tests leverage the hadoop-provided `MiniDFSCluster`, S3 tests deploys a MinIO instance in docker. I'm not sure if there's a good way to setup a light-weighted testing purpose google storage service. If it requires too much efforts, I'd be fine with adding only the GCS specific unit tests, leaving the IT cases / E2E tests as a separate follow-up effort. PS: `AbstractRecoverableWriterTest` is probably also related, in addition to the two classes you've mentioned. PPS: We may also consider updating the documentation in a separate follow-up PR. For a major feature like this, we might want to involve some product specialists for the documentation review. It would help simplify the review process if the engineering and documentation changes are separated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
