Hi Shawn, If you're asking me if Spark SQL should optimize such queries, I don't know.
If you're asking me if it's possible to convince Spark SQL to do so, I'd say, sure, it is. Write your optimization rule and attach it to Optimizer (using extraOptimizations extension point). Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Thu, Apr 27, 2017 at 3:22 PM, Lavelle, Shawn <shawn.lave...@osii.com> wrote: > Hi Jacek, > > > > I know that it is not currently doing so, but should it be? The > algorithm isn’t complicated and could be applied to both OR and AND logical > operators with comparison operators as children. > My users write programs to generate queries that aren’t checked for > this sort of thing. We’re probably going to write our own > org.apache.spark.sql.catalyst.rules.Rule to handle it. > > ~ Shawn > > > > *From:* Jacek Laskowski [mailto:ja...@japila.pl] > *Sent:* Wednesday, April 26, 2017 2:55 AM > *To:* Lavelle, Shawn <shawn.lave...@osii.com> > *Cc:* user <user@spark.apache.org> > *Subject:* Re: Spark-SQL Query Optimization: overlapping ranges > > > > explain it and you'll know what happens under the covers. > > > > i.e. Use explain on the Dataset. > > > > Jacek > > > > On 25 Apr 2017 12:46 a.m., "Lavelle, Shawn" <shawn.lave...@osii.com> > wrote: > > Hello Spark Users! > > Does the Spark Optimization engine reduce overlapping column ranges? > If so, should it push this down to a Data Source? > > Example, > > This: Select * from table where col between 3 and 7 OR col between 5 > and 9 > > Reduces to: Select * from table where col between 3 and 9 > > > > > > Thanks for your insight! > > > ~ Shawn M Lavelle > > > > > > *Shawn* *Lavelle* > Software Development > > 4101 Arrowhead Drive > Medina, Minnesota 55340-9457 > Phone: 763 551 0559 <(763)%20551-0559> > Fax: 763 551 0750 <(763)%20551-0750> > *Email:* *shawn.lave...@osii.com <shawn.lave...@osii.com>* > *Website:* *www.osii.com* <http://www.osii.com> > > > >
