lower and upper offset not working in spark with mysql database
Hi all! I am trying to read records from offset 100 to 110 from a table using following piece of code val sc = new SparkContext(new SparkConf().setAppName(SparkJdbcDs).setMaster(local[*])) val sqlContext = new SQLContext(sc) val options = new HashMap[String, String]() options.put(driver, com.mysql.jdbc.Driver) options.put(url, jdbc:mysql://***:3306/temp?user=password=) options.put(dbtable, tempTable) options.put(lowerBound, 100) options.put(upperBound, 110) options.put(numPartitions, 1) sqlContext.load(jdbc, options) but this returns all the records instead of only 10 records -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/lower-and-upper-offset-not-working-in-spark-with-mysql-database-tp23635.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
RE: lower and upper offset not working in spark with mysql database
I think you should mention partitionColumn like below and the Colum type should be numeric. It works for my case. options.put(partitionColumn, revision); Thanks, Manohar From: Hafiz Mujadid [via Apache Spark User List] [mailto:ml-node+s1001560n23635...@n3.nabble.com] Sent: Monday, July 6, 2015 10:56 AM To: Manohar Reddy Subject: lower and upper offset not working in spark with mysql database Hi all! I am trying to read records from offset 100 to 110 from a table using following piece of code val sc = new SparkContext(new SparkConf().setAppName(SparkJdbcDs).setMaster(local[*])) val sqlContext = new SQLContext(sc) val options = new HashMap[String, String]() options.put(driver, com.mysql.jdbc.Driver) options.put(url, jdbc:mysql://***:3306/temp?user=password=) options.put(dbtable, tempTable) options.put(lowerBound, 100) options.put(upperBound, 110) options.put(numPartitions, 1) sqlContext.load(jdbc, options) but this returns all the records instead of only 10 records If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/lower-and-upper-offset-not-working-in-spark-with-mysql-database-tp23635.html To start a new topic under Apache Spark User List, email ml-node+s1001560n1...@n3.nabble.commailto:ml-node+s1001560n1...@n3.nabble.com To unsubscribe from Apache Spark User List, click herehttp://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_codenode=1code=bWFub2hhci5yZWRkeUBoYXBwaWVzdG1pbmRzLmNvbXwxfDM0NDk1ODYyNg==. NAMLhttp://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml Happiest Minds Disclaimer This message is for the sole use of the intended recipient(s) and may contain confidential, proprietary or legally privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the original intended recipient of the message, please contact the sender by reply email and destroy all copies of the original message. Happiest Minds Technologies http://www.happiestminds.com -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/lower-and-upper-offset-not-working-in-spark-with-mysql-database-tp23635p23637.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: lower and upper offset not working in spark with mysql database
thanks On Mon, Jul 6, 2015 at 10:46 AM, Manohar753 [via Apache Spark User List] ml-node+s1001560n23637...@n3.nabble.com wrote: I think you should mention partitionColumn like below and the Colum type should be numeric. It works for my case. options.put(partitionColumn, revision); Thanks, Manohar *From:* Hafiz Mujadid [via Apache Spark User List] [mailto:ml-node+[hidden email] http:///user/SendEmail.jtp?type=nodenode=23637i=0] *Sent:* Monday, July 6, 2015 10:56 AM *To:* Manohar Reddy *Subject:* lower and upper offset not working in spark with mysql database Hi all! I am trying to read records from offset 100 to 110 from a table using following piece of code val sc = new SparkContext(new SparkConf().setAppName(SparkJdbcDs).setMaster(local[*])) val sqlContext = new SQLContext(sc) val options = new HashMap[String, String]() options.put(driver, com.mysql.jdbc.Driver) options.put(url, jdbc:mysql://***:3306/temp?user=password=) options.put(dbtable, tempTable) options.put(lowerBound, 100) options.put(upperBound, 110) options.put(numPartitions, 1) sqlContext.load(jdbc, options) but this returns all the records instead of only 10 records -- *If you reply to this email, your message will be added to the discussion below:* http://apache-spark-user-list.1001560.n3.nabble.com/lower-and-upper-offset-not-working-in-spark-with-mysql-database-tp23635.html To start a new topic under Apache Spark User List, email [hidden email] http:///user/SendEmail.jtp?type=nodenode=23637i=1 To unsubscribe from Apache Spark User List, click here. NAML http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml -- Happiest Minds Disclaimer This message is for the sole use of the intended recipient(s) and may contain confidential, proprietary or legally privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the original intended recipient of the message, please contact the sender by reply email and destroy all copies of the original message. Happiest Minds Technologies http://www.happiestminds.com -- -- If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/lower-and-upper-offset-not-working-in-spark-with-mysql-database-tp23635p23637.html To unsubscribe from lower and upper offset not working in spark with mysql database, click here http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_codenode=23635code=aGFmaXptdWphZGlkMDBAZ21haWwuY29tfDIzNjM1fC05MjEzOTMxMTE= . NAML http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml -- Regards: HAFIZ MUJADID -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/lower-and-upper-offset-not-working-in-spark-with-mysql-database-tp23635p23638.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
