leezu opened a new issue #18233: URL: https://github.com/apache/incubator-mxnet/issues/18233
Bringing some discussion from https://github.com/apache/incubator-mxnet/pull/18222#pullrequestreview-404567393 to a separate issue. Main goal is to raise awareness that the current state is not optimal and partially broken. As there have been veto's on workarounds for the broken parts, let's raise awareness so that going forward the community may collaborate on improving the state more efficiently. On a high level I'm calling out that the internal CD S3 bucket used by the CD is actually internal and we can change the object structure as we like over time. The only "archived" artifacts should be the ones pushed to public bucket, on which of course we like to keep the structure stable. Secondly, object expiration. No need to keep temporary files in a private bucket around forever. On a more detailed point, we need to discuss the folder structure used by CD and / or the atcual build jobs of the CD. Currently CD builds for each of: `{git_sha}/{libtype}/{os}/{variant}/` where git sha is the git commit and variant refers to the cpu/gpu build variants. ### OS Currently CD uses the `/etc/os-release` of the This broke AMI running the stage of the CD as identifier of the s3 folder. That's useless, as we may build artifacts for different targets on the same AMI (cross-compilation). In fact there is no need that the GPU and CPU stages run the same AMI.. As this feature is broken and is unused (only build for one target at this time), we should delete it to keep the code simple. Instead we can use a cross-compilation triplet here. See https://github.com/apache/incubator-mxnet/pull/18222#pullrequestreview-404567393 for more details. ### libtype Both `libtype=static` and `libtype=dynamic` build the dynamic `libmxnet`, the only difference is the number of dependencies statically linked. But in fact `libtype=dynamic` artifacts are not used anywhere. `libtype=static` is fully sufficient and we can reduce complexity and costs while achieving the same goal by removing unused `libtype=dynamic`. The goal here is to keep things simple and avoid unnecessary complexity. The idea behind the unused `libtype=dynamic` pipeline is to only rely on dynamic linking and not link any dependencies statically. This is great goal in principle and exactly what is done by Linux distributions such as Debian and Ubuntu. We can support this goal by working with the distributions and getting MXNet packaged there. But I highly doubt that we like to do this in a nighlty fashion on our CD. For that, the libtype=static` version of our dynamic library is fully sufficient. By removing unnecessary and / or unused things, we can keep the code simple and easier to maintain. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
