nicknezis commented on pull request #3725: URL: https://github.com/apache/incubator-heron/pull/3725#issuecomment-968204398
Yes, the case where we mount an existing PVC, this would be a shared PVC accessed by all pods. ReadWriteMany or a read only mount to share static files. One example is you could mount a read only PVC with files such as certificates and certificate authorities. You can also mount a shared drive for large file access. There are cases in which streaming data is smaller references to larger files. The bolts would have access to the shared volumes to access larger files. The bolts could do file scan actions for specific files based on the metadata only stream. The case for the dynamic per-pod mounts using the PersistentVolumeClaim Template would be for local checkpointing and log file storage. Currently in each pod there is a `log-files` folder in each container that writes the Spout/Bolt logs to the container's built in storage. With the dynamic PVC mount, one could configure to use a system like Rook/Ceph to provide dynamic storage that does not use the Docker volume which might be more limited. If there is a better suggestion for the CLI parameters, I'm cool with that. I was merely using the Spark design as a starting point. I see the value in how they structured the subsets of config under `mount` and `options` that supports handling the different storage types (i.e. `nfs`, `pvc`, `hostPath`, etc). But we could totally have a more explicit set of config. I think it's my fault for discussing both the static mount and dynamic mont. Hopefully I didn't confuse things too much. I really was looking to mirror what Spark provided in terms of the functionality. I think their `OnDemand` and `local` are the only cases which get the per pod specific storage. The other cases, the pods share the same storage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
