Hi! I raised a ticket on parametrisable output metadata path https://issues.apache.org/jira/browse/SPARK-43152. I am going to raise a PR against it and I realised, that this relatively simple change impacts on method hasMetadata(path), that would have a new meaning if we can define custom path for metadata of output files. Can you please share your opinion on how the custom output metadata path can impact on design of structured streaming? E.g. I can see one case when I set a parameter of output metadata path, run a job on output path A, stop the job, change the output path to B and hasMetadata works well. If you have any corner case in mind where the parametrised output metadata path can break something please describe it.
-- Kind regards/ Pozdrawiam, Wojciech Indyk
