HI, Could anybody provide inputs if they have came across similar issue @Rishitesh Could you provide if any sample code to use JdbcRDDSuite
Regards, Satish Chandra On Wed, Sep 23, 2015 at 5:14 PM, Rishitesh Mishra <rishi80.mis...@gmail.com> wrote: > I am using Spark 1.5. I always get count = 100, irrespective of num > partitions. > > On Wed, Sep 23, 2015 at 5:00 PM, satish chandra j < > jsatishchan...@gmail.com> wrote: > >> HI, >> Currently using Spark 1.2.2, could you please let me know correct results >> output count which you got it by using JdbcRDDSuite >> >> Regards, >> Satish Chandra >> >> On Wed, Sep 23, 2015 at 4:02 PM, Rishitesh Mishra < >> rishi80.mis...@gmail.com> wrote: >> >>> Which version of Spark you are using ?? I can get correct results using >>> JdbcRDD. Infact there is a test suite precisely for this (JdbcRDDSuite) >>> . >>> I changed according to your input and got correct results from this test >>> suite. >>> >>> On Wed, Sep 23, 2015 at 11:00 AM, satish chandra j < >>> jsatishchan...@gmail.com> wrote: >>> >>>> HI All, >>>> >>>> JdbcRDD constructor has following parameters, >>>> >>>> *JdbcRDD >>>> <https://spark.apache.org/docs/1.2.0/api/java/org/apache/spark/rdd/JdbcRDD.html#JdbcRDD(org.apache.spark.SparkContext,%20scala.Function0,%20java.lang.String,%20long,%20long,%20int,%20scala.Function1,%20scala.reflect.ClassTag)>* >>>> (SparkContext >>>> <https://spark.apache.org/docs/1.2.0/api/java/org/apache/spark/SparkContext.html> >>>> sc, >>>> scala.Function0<java.sql.Connection> getConnection, String sql, *long >>>> lowerBound, >>>> long upperBound, int numPartitions*, >>>> scala.Function1<java.sql.ResultSet,T >>>> <https://spark.apache.org/docs/1.2.0/api/java/org/apache/spark/rdd/JdbcRDD.html>> >>>> mapRow, >>>> scala.reflect.ClassTag<T >>>> <https://spark.apache.org/docs/1.2.0/api/java/org/apache/spark/rdd/JdbcRDD.html> >>>> > evidence$1) >>>> >>>> where the below parameters *lowerBound* refers to Lower boundary of >>>> entire data, *upperBound *refers to Upper boundary of entire data and >>>> *numPartitions >>>> *refer to Number of partitions >>>> >>>> Source table to which JbdcRDD is fetching data from Oracle DB has more >>>> than 500 records but its confusing when I tried several executions by >>>> changing "numPartitions" parameter >>>> >>>> LowerBound,UpperBound,numPartitions: Output Count >>>> >>>> 0 ,100 ,1 : 100 >>>> >>>> 0 ,100 ,2 : 151 >>>> >>>> 0 ,100 ,3 : 201 >>>> >>>> >>>> Please help me in understanding the why Output count is 151 if >>>> numPartitions is 2 and Output count is 201 if numPartitions is 3 >>>> >>>> Regards, >>>> >>>> Satish Chandra >>>> >>> >>> >> >
