jihoonson commented on pull request #9898: URL: https://github.com/apache/druid/pull/9898#issuecomment-650608153
@FrankChen021 thank you for the explanation. > Hi @jihoonson , s3 extension does not work with aliyun OSS although OSS provides compatibility with s3. I tested to utilize OSS with s3 extension in the past, but the `putObject` returned HTTP 400 error, saying there were some unsupported HEADERS in the request generated by s3 SDK. I didn't explore that problem very deeply, I guess the latest s3 SDK adds some new HEADERS which are not compatible in OSS. Do you mean the OSS APIs are not actually compatible to S3 APIs? I think this is the most important question to decide what we should do here, and so we should figure it out before we proceed. Do you remember what the exact error was? Probably it would be one of the errors listed in https://www.alibabacloud.com/help/doc-detail/32005.htm. If you don't remember, since you seem to already have a testing environment, would you please test it again if it's not that hard for you? > the APIs between OSS and s3 are very similar, even their names. but I don't think it's a good idea to use one extension to support both OSS and s3, or use some abstraction method to extract the same code between s3 and OSS. Actually, this kind of abstraction has been done by deep storage framework, so every implementation interacts with its underlying storage API. As s3 and OSS are provided by two companies, it's better to provide two extensions respectively so that they can evolve independently. I agree abstraction layer is not a good idea. Since an extension cannot depend on another, an abstraction layer beween s3 and OSS requires to be in the core which I don't think is an option. I haven't looked at the PR very closely, but it looks almost identical to the s3 extension. An obvious downside in developing a new extension is the increasing maintenance burden; when we modify the s3 extension, we have to check the OSS extension if it's reasonable to apply the same change, or vice versa. However, there is a problem in using s3 extension for OSS as well. Whenever we modify the s3 extension, we have to check the change is causing any issue in using OSS. So, I cannot tell what the right way is until we figure out the compatibility level between OSS and S3. It makes sense to me to develop a new extension if one of the following statements is true. - OSS APIs are not actually compatible to S3 APIs. They look compatible based on their docs (with small differences in permission system), but it might not be true based on your comment above. - The APIs are compatible in most cases, but there are some exceptions in particular APIs which we are using. - The APIs are completely compatible now, but OSS is planning to stop providing s3-compatibility. Is there anything about this you are aware of? The first 2 statements can be verified through some testings. For the last statement, I couldn't find anything in 1 min with my googling skill. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
