Sreelal S L created SPARK-17621:

             Summary: Accumulator value is doubled when using 
                 Key: SPARK-17621
             Project: Spark
          Issue Type: Bug
          Components: Scheduler, SQL
    Affects Versions: 2.0.0
         Environment: Development environment. (Eclipse . Single process) 
            Reporter: Sreelal S L
            Priority: Minor

We are tracing the records read by our source using an accumulator.  We do a 
orderBy on the Dataframe before the output operation. When the job is 
completed, the accumulator values is becoming double of the expected value . . 
Below is the sample code i ran . 


 val sqlContext = SparkSession.builder() 
false).config("spark.sql.warehouse.dir", "file:///C:/Test").master("local[*]")
    val sc = sqlContext.sparkContext
    val accumulator1 = sc.accumulator(0, "accumulator1")

    val usersDF ="C:\\users.json") //  single row 
{"name":"sreelal" ,"country":"IND"}
    val usersDFwithCount = => { accumulator1 += 1; x });
    val counterDF = sqlContext.createDataFrame(usersDFwithCount, 
    val oderedDF = counterDF.orderBy("name")

    val collected = oderedDF.collect()
    collected.foreach { x => println(x) }

    println("accumulator1 : " + accumulator1.value)


I have only one row in the users.json file.  I expect accumulator1 to have 
value 1. But its coming as 2. 

In the Spark Sql UI , i see two jobs getting generated for the same. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to