I wonder if there's some recommended method to convert in memory pyarrow.Table (or pyarrow.BatchRecord) to pyspark.Dataframe without using pandas ? My motivation is about converting nested data (like List[int]) that have an efficient representation in pyarrow which is not possible with Pandas (I don't want to pass by python list of int ...).
Thanks in advance ! Artem
