[
https://issues.apache.org/jira/browse/SPARK-23969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-23969.
----------------------------------
Resolution: Incomplete
> Using FPGrowth with PipelinedRDD gives EOF Error
> ------------------------------------------------
>
> Key: SPARK-23969
> URL: https://issues.apache.org/jira/browse/SPARK-23969
> Project: Spark
> Issue Type: Bug
> Components: MLlib, Spark Submit
> Affects Versions: 2.0.2
> Environment: Spark 2.0.2
> Python2.6 (This is due to a slight issue with the Altiscale environment we
> are using. We will be moving to 3.5)
> Driver Memory = 32GB
> 5 nodes with 4 cores each and 32GB memory
> Shuffle Service = False
> Dynamic Allocation = False
> Cross Join = True
> Reporter: Srinivasan
> Priority: Major
> Labels: bulk-closed
> Attachments: EOFError.docx
>
>
> I am trying to find association rule on a data set with 27 million rows and 9
> columns. The data is stored in a hive table and loaded into an RDD.
> I am not using the collect function as I keep getting out of memory error. I
> understand that FPGrowth needs a list of transactions. Which means I need to
> convert my dataframe into a list of lists. But I keep getting serializer.py
> EOF error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
