Hi all! Is there any info on how to scale PIO to multiple nodes? I’ve gone through a lot of the docs on the site and haven’t found anything. I’ve tested PIO running with HBASE and ES for metadata and events, and with using just ES for both (my preference thusfar) and have my models on local storage. Would scaling simply be a matter of deploying clustered ES, and then finding some way to share my model storage, such as NFS or HDFS? The question then is what (if anything) has to be done for the nodes to “know” about changes on other nodes. For example, if the model gets trained on node A does node B automatically know about that?
I hope that makes sense. I’m coming to PIO with no prior experience for the underlying apache bits (spark, hbase / hdfs, etc) so there’s likely things I’m not considering. Any help / docs / guidance is appreciated. Thanks! Adam
