[jira] (SPARK-19401) Transformation of one RDD inside of other transformations

[Chetan Panchal (JIRA)]

Title: Message Title

Chetan Panchal created an issue   Spark / SPARK-19401 Transformation of one RDD inside of other transformations Issue Type: Question Assignee: Unassigned Created: 30/Jan/17 10:20 Priority: Major Reporter: Chetan Panchal Hi, I am new in spark. I want to update one RDD Map key by matching in another RDD. When i am using .take(100) or collect(), this is working fine but it is taking so much time. But when i am using .map(), it is giving me error. org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 381.0 failed 4 times, most recent failure: Lost task 0.3 in stage 381.0 (TID 707, 192.168.200.82): org.apache.spark.SparkException: RDD transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(x => rdd2.values.count() * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more information, see SPARK-5063 . Add Comment   This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)