Hey Friends, Is there an easy way of figuring out whats being pull into the task context? I’ve been getting the following message which I suspect means I’ve unintentional caught some large objects but figuring out what those objects are is stumping me.
19/04/23 13:52:13 WARN org.apache.spark.internal.Logging$class TaskSetManager: Stage 152 contains a task of very large size (12747 KB). The maximum recommended task size is 100 KB Cheers Andrew
