[jira] [Updated] (GEODE-120) RDD.saveToGemfire() can not handle big dataset (1M entries per partition)
[ https://issues.apache.org/jira/browse/GEODE-120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anthony Baker updated GEODE-120: Fix Version/s: 1.0.0-alpha1 > RDD.saveToGemfire() can not handle big dataset (1M entries per partition) > - > > Key: GEODE-120 > URL: https://issues.apache.org/jira/browse/GEODE-120 > Project: Geode > Issue Type: Sub-task > Components: core, extensions >Affects Versions: 1.0.0-incubating >Reporter: Qihong Chen >Assignee: Qihong Chen > Fix For: 1.0.0-alpha1 > > Original Estimate: 48h > Remaining Estimate: 48h > > the connector use single region.putAll() call to save each RDD partition. But > putAll() doesn't handle big dataset well (such as 1M record). Need to split > the dataset into smaller chunks, and invoke putAll() for each chunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEODE-120) RDD.saveToGemfire() can not handle big dataset (1M entries per partition)
[ https://issues.apache.org/jira/browse/GEODE-120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qihong Chen updated GEODE-120: -- Summary: RDD.saveToGemfire() can not handle big dataset (1M entries per partition) (was: RDD.saveToGemfire() can not handle big dataset (1M record per partition)) RDD.saveToGemfire() can not handle big dataset (1M entries per partition) - Key: GEODE-120 URL: https://issues.apache.org/jira/browse/GEODE-120 Project: Geode Issue Type: Sub-task Components: core, extensions Affects Versions: 1.0.0-incubating Reporter: Qihong Chen Assignee: Qihong Chen Original Estimate: 48h Remaining Estimate: 48h the connector use single region.putAll() call to save each RDD partition. But putAll() doesn't handle big dataset well (such as 1M record). Need to split the dataset into smaller chunks, and invoke putAll() for each chunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
