Jatin Puri created SPARK-24784:
----------------------------------
Summary: Retraining (each document as separate file) creates OOME
Key: SPARK-24784
URL: https://issues.apache.org/jira/browse/SPARK-24784
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.3.1
Environment: Spark 2.3.1
JDK8
SBT 1.1.6
Ubuntu 18.04
Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
Reporter: Jatin Puri
This looks like a bug. The reproducible can be found at:
[https://github.com/purijatin/spark-retrain-bug]
Information also available at:
[http://apache-spark-developers-list.1001551.n3.nabble.com/Retraining-with-each-document-as-separate-file-creates-OOME-td24335.html]
*Issue:*
The program takes input a set of individual files (json) and each file contains
a single document with the same json schema as others.
The spark program computes the tf-idf of the documents (Tokenizer -> Stopword
remover -> stemming -> tf -> tfidf).
Once the training is complete, it unpersists and restarts the operation again.
When run with `-Xmx1500M` memory, it fails with OOME after about 10
iterations.
*Temporary Fix:*
When all the input documents are merged to a single file, then the issue is no
longer found.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
