*Correction, 60C2 * 3*
---------- Forwarded message ---------- From: Aakash Basu <aakash.spark....@gmail.com> Date: Mon, Jun 18, 2018 at 4:15 PM Subject: StackOverFlow ERROR - Bulk interaction for many columns fail To: user <user@spark.apache.org> Hi, When doing bulk interaction on around 60 columns, I want 3 columns to be created out of each one of them, since it has a combination of 3, then it becomes 60C2 * 3, which creates a lot of columns. So, for a lesser than 50 - 60 columns, even though it takes time, it still works fine, but, for a little larger number of columns, it throws this error - File "/usr/local/lib/python3.5/dist-packages/backend/feature_ > extraction/cont_bulk_interactions.py", line 100, in bulktransformer_pairs > df = df.withColumn(col_name, each_op(df[var_2], df[var_1])) > File > "/appdata/spark-2.3.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/dataframe.py", > line 1849, in withColumn > File "/appdata/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0. > 10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__ > File > "/appdata/spark-2.3.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/utils.py", > line 63, in deco > File "/appdata/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0. > 10.7-src.zip/py4j/protocol.py", line 328, in get_return_value > py4j.protocol.Py4JJavaError: An error occurred while calling > o26887.withColumn. > : java.lang.StackOverflowError > at scala.collection.generic.GenericTraversableTemplate$ > class.genericBuilder(GenericTraversableTemplate.scala:70) > at scala.collection.AbstractTraversable.genericBuilder(Traversable. > scala:104) > at scala.collection.generic.GenTraversableFactory$ > GenericCanBuildFrom.apply(GenTraversableFactory.scala:57) > at scala.collection.generic.GenTraversableFactory$ > GenericCanBuildFrom.apply(GenTraversableFactory.scala:52) > at scala.collection.TraversableLike$class.builder$ > 1(TraversableLike.scala:229) > at scala.collection.TraversableLike$class.map( > TraversableLike.scala:233) > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > What to do? Thanks, Aakash.
