Jatin Puri created SPARK-24784:

             Summary: Retraining (each document as separate file) creates OOME
                 Key: SPARK-24784
                 URL: https://issues.apache.org/jira/browse/SPARK-24784
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.3.1
         Environment: Spark 2.3.1
SBT 1.1.6
Ubuntu 18.04
Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
            Reporter: Jatin Puri

This looks like a bug. The reproducible can be found at: 

Information also available at: 

The program takes input a set of individual files (json) and each file contains 
a single document with the same json schema as others.
The spark program computes the tf-idf of the documents  (Tokenizer -> Stopword 
remover -> stemming -> tf -> tfidf).
Once the training is complete, it unpersists and restarts the operation again.
When run with  `-Xmx1500M` memory, it fails with OOME after about 10 
*Temporary Fix:*
When all the input documents are merged to a single file, then the issue is no 
longer found. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to