Hello team I was exploring feature transformation exposed via Mllib on nested dataset, and encountered an error while applying any transformer to a column with dot notation naming. I thought of raising a ticket on spark https://issues.apache.org/jira/browse/SPARK-48463, where I have mentioned the entire scenario.
I wanted to get suggestions on what would be the best way to solve the problem while using the dot notation. One workaround is to use`_` while flattening the dataframe, but that would mean having an additional overhead to convert back to `.` (dot notation ) since that’s the convention for our other flattened data. I would be happy to make a contribution to the code if someone can shed some light on how this could be solved. -- Thanks and Regards, Chhavi Bansal
