[
https://issues.apache.org/jira/browse/SPARK-12927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-12927.
-------------------------------
Resolution: Not A Problem
You have copied the contents of a DataFrame to the driver with collect(). That
copy won't change just because an external system changes. The results are
immutable, though, in theory might change if the resulting DataFrame is
recomputed and the underlying data has changed. However, you don't know if/when
this happens. Really, you need to make a new DataFrame.
> Dataframe doesn't display the current status on Aerospike.
> ----------------------------------------------------------
>
> Key: SPARK-12927
> URL: https://issues.apache.org/jira/browse/SPARK-12927
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.5.1
> Reporter: Sasi
>
> Hi,
> I'm facing with issue on my environment when my dataframe doesn't display the
> real status of my Aerospike bin.
> I have the following code:
> {code}
> SparkContextFunctions sparkContextFunctions = new
> SparkContextFunctions(spark_context);
> String aeroSpikeAddress = host+port;
> String query = "select * from bin";
> dataFrame = sparkContextFunctions.aeroSInput(aeroSpikeAddress,
> query, sqlCtx, 6);
> dataFrame .registerTempTable("testTbl");
> dataFrame .persist(StorageLevel.MEMORY_ONLY());
> {code}
> After create the tmpTbl I do the following:
> {code}
> subscribersDataFrame.where(whereClause).collect();
> {code}
> At the first time I get the same values of the Aerospike bin, but if I add or
> delete row I don't get the same status.
> If I drop the table and create it again then I get the right status.
> Thanks.
> Sasi
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
