My bad !!! Yea you are right.But now the problem is when I get DataFrame by
using below code and get JavaRDD from it , its number of partitions is 1 and
its of type MapPartitionsRDD.
String sql = "select simulationUUID,stockReturn from STOCKSIMULATIONRETURNSVAL
where businessDate = ? and symbol = ?";DataFrame df
=jic.fromCache(PARTITIONED_CACHE_NAME).sql(sql, businessDate,stock);
df.javaRDD();
This is causing performance issues for me as I have only 1 partition and my
reduceByKey method is not performing in desired way.
Regards,Vij
On Friday, April 29, 2016 6:11 PM, Vladimir Ozerov <[email protected]>
wrote:
Hi Vij,
I see method "getPartitions" in IgniteRDD, not "getNumPartitions". Please
confirm that we are talking about the same thing.
Anyway, logic of this method is extremely straightforward - it simply call
Ignite.affinity("name_of_your_cache").partitions() method, so it should return
actual number of partitions. "getPartitions" returns array, could you please
show is printed to the console from your code?
Vladimir.
On Fri, Apr 29, 2016 at 3:10 PM, vijayendra bhati <[email protected]>
wrote:
Yes its Spark RDD's standard method, but it has been overridden in IgniteRDD.
Regards,Vij
On Friday, April 29, 2016 5:25 PM, Vladimir Ozerov <[email protected]>
wrote:
Hi Vij,
I am not quite uderstand where does method "getNumPartitions" came from. Is it
on standard Spark API? I do not see it on org.apache.spark.api.java.JavaRDD
class.
Vladimir.
On Fri, Apr 29, 2016 at 7:50 AM, vijayendra bhati <[email protected]>
wrote:
Hi Val,
I am creating DataFrame using below code -
public DataFrame getStockSimulationReturnsDataFrame(LocalDate
businessDate,String stock){ /* * If we use sql query , we are assuming that
data is in cache. * */ String sql = "select simulationUUID,stockReturn from
STOCKSIMULATIONRETURNSVAL where businessDate = ? and symbol = ?"; DataFrame df
=jic.fromCache(PARTITIONED_CACHE_NAME).sql(sql, businessDate,stock); return df;
}
And to check partitions I am doing -
private JavaRDD<Row> getSimulationsForStock(String stock,LocalDate
businessDate) { DataFrame df =
StockSimulationsReaderFactory.getStockSimulationStore(jsc, businessDate,
businessDate).getStockSimulationReturnsDataFrame(businessDate, stock);
System.out.println("^^^^^^^^"+df.javaRDD().getNumPartitions()); return
df.javaRDD(); }
Regards,Vij
On Friday, April 29, 2016 3:09 AM, vkulichenko
<[email protected]> wrote:
Hi Vij,
How do you check the number of partitions and what are you trying to
achieve? Can you show the code?
-Val
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Number-of-partitions-of-IgniteRDD-tp4644p4671.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
