Hello everyone, I am new in Pyspark and i am facing a problem in casting some values in DecimalType. To clarify my question i present an example.
i have a dataframe in which i store my data which are some trajectories the dataframe looks like *Id | Trajectory* id1 | [ [x1, y1, t1], [x2, y2, t2], .......[xn, yn, tn] ] id2 | [ [x1, y1, t1], [x2, y2, t2], .......[xn, yn, tn] ] So for each ID i have a trajectory after applying group by in a previous dataframe. x= lon y=lat t=time (in seconds). When i store the data in the dataframe the time is displayed in scientific notation. Is there any way to convert the t in DecimalType(10,0) ? I tried to convert it in the beggining but i get an error when i try to store the trajectories because the column Trajectory is ArrayType(ArrayType(FloatType() ) ).
