Hi Martin, Agree, if you don't need the other features of MLFlow then it is likely overkill.
Cheers, Sonal https://github.com/zinggAI/zingg On Mon, Oct 25, 2021 at 4:06 PM <mar...@wunderlich.com> wrote: > Hi Sonal, > > Thanks a lot for this suggestion. I presume it might indeed be possible to > use MLFlow for this purpose, but at present it seems a bit too much to > introduce another framework only for storing arbitrary meta-data with > trained ML pipelines. I was hoping there might be a way to do this natively > in Spark ML. Otherwise, I'll just create a wrapper class for the trained > models. > > Cheers, > > Martin > > > > Am 2021-10-24 21:16, schrieb Sonal Goyal: > > Does MLFlow help you? https://mlflow.org/ > > I don't know if ML flow can save arbitrary key-value pairs and associate > them with a model, but versioning and evaluation etc are supported. > > Cheers, > Sonal > https://github.com/zinggAI/zingg > > > On Wed, Oct 20, 2021 at 12:59 PM <mar...@wunderlich.com> wrote: > > Hello, > > This is my first post to this list, so I hope I won't violate any > (un)written rules. > > I recently started working with SparkNLP for a larger project. SparkNLP in > turn is based Apache Spark's MLlib. One thing I found missing is the > ability to store custom parameters in a Spark pipeline. It seems only > certain pre-configured parameter values are allowed (e.g. "stages" for the > Pipeline class). > > IMHO, it would be handy to be able to store custom parameters, e.g. for > model versions or other meta-data, so that these parameters are stored with > a trained pipeline, for instance. This could also be used to include > evaluation results, such as accuracy, with trained ML models. > > (I also asked this on Stackoverflow, but didn't get a response, yet: > https://stackoverflow.com/questions/69627820/setting-custom-parameters-for-a-spark-mllib-pipeline > ) > > Would does the community think about this proposal? Has it been discussed > before perhaps? Any thoughts? > > Cheers, > > Martin > >
