Thanks Michael, much appreciated! Nothing should be held in memory for a query like this (other than a single count per partition), so I don't think that is the problem. There is likely an error buried somewhere.
For your above comments - I don't get any error but just get the NULL as return value. I have tried digging deeper in the logs etc but couldn't spot anything. Is there any other suggestions to spot such buried errors? Thanks, Dhaval On Mon, Aug 24, 2015 at 6:38 PM, Michael Armbrust <mich...@databricks.com> wrote: > Much appreciated! I am not comparing with "select count(*)" for >> performance, but it was one simple thing I tried to check the performance >> :). I think it now makes sense since Spark tries to extract all records >> before doing the count. I thought having an aggregated function query >> submitted over JDBC/Teradata would let Teradata do the heavy lifting. >> > > We currently only push down filters since there is a lot of variability in > what types of aggregations various databases support. You can manually > pushdown whatever you want by replacing the table name with a subquery > (i.e. "(SELECT ... FROM ...)") > > - How come my second query for (5B) records didn't return anything >> even after a long processing? If I understood correctly, Spark would try to >> fit it in memory and if not then might use disk space, which I have >> available? >> > > Nothing should be held in memory for a query like this (other than a > single count per partition), so I don't think that is the problem. There > is likely an error buried somewhere. > > >> - Am I supposed to do any Spark related tuning to make it work? >> >> My main need is to access data from these large table(s) on demand and >> provide aggregated and calculated results much quicker, for that I was >> trying out Spark. Next step I am thinking to export data in Parque files >> and give it a try. Do you have any suggestions for to deal with the problem? >> > > Exporting to parquet will likely be a faster option that trying to query > through JDBC, since we have many more opportunities for parallelism here. >
