i used spark 2.0.1 and work pypsaprk.sql dataframe
lower = arguments["lower"] lower_udf = udf(lambda x: lower if x<lower else x, IntegerType()) data.withColumn(col_name,lower_udf(col_name)).show()
i used spark 2.0.1 and work pypsaprk.sql dataframe
lower = arguments["lower"] lower_udf = udf(lambda x: lower if x<lower else x, IntegerType()) data.withColumn(col_name,lower_udf(col_name)).show()