[ https://issues.apache.org/jira/browse/SPARK-12521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15071347#comment-15071347 ]
Xiu (Joe) Guo commented on SPARK-12521: --------------------------------------- In 1.5.2 {code}sqlContext.load(){code} is deprecated, but I can still reproduce with:{code}sqlContext.read.jdbc(){code} I don't think it is the size of your numbers. I can reproduce with small integers given as lowerBound/upperBound with my setup. Can you maybe try adding "L" at the end of your number to verify that it still gives wrong results? I think the problem is the lowerBound and upperBound are not honored here, Spark just retrieves every row instead of 1001 rows bounded in your case. > DataFrame Partitions in java does not work > ------------------------------------------ > > Key: SPARK-12521 > URL: https://issues.apache.org/jira/browse/SPARK-12521 > Project: Spark > Issue Type: Bug > Components: Java API > Affects Versions: 1.5.2 > Reporter: Sergey Podolsky > > Hello, > Partition does not work in Java interface of the DataFrame: > {code} > SQLContext sqlContext = new SQLContext(sc); > Map<String, String> options = new HashMap<>(); > options.put("driver", ORACLE_DRIVER); > options.put("url", ORACLE_CONNECTION_URL); > options.put("dbtable", > "(SELECT * FROM JOBS WHERE ROWNUM < 10000) tt"); > options.put("lowerBound", "2704225000"); > options.put("upperBound", "2704226000"); > options.put("partitionColumn", "ID"); > options.put("numPartitions", "10"); > DataFrame jdbcDF = sqlContext.load("jdbc", options); > List<Row> jobsRows = jdbcDF.collectAsList(); > System.out.println(jobsRows.size()); > {code} > gives 9999 while expected 1000. Is it because of big decimal of boundaries or > partitioins does not work at all in Java? > Thanks. > Sergey -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org