Hi Akhil
I guess it skipped my attention. I would definitely give it a try.
While I would still like to know what is the issue with the way I have
created schema?
On Tue, Feb 24, 2015 at 4:35 PM, Akhil Das
wrote:
> Did you happen to have a look at
> https://spark.apache.org/docs/latest/sql-pro
ields.
On Wed, Feb 25, 2015 at 9:45 AM, anamika gupta
wrote:
> Hi Akhil
>
> I guess it skipped my attention. I would definitely give it a try.
>
> While I would still like to know what is the issue with the way I have
> created schema?
>
> On Tue, Feb 24, 2015 at 4:35 PM,
7).trim, r(18).trim,
> r(19).trim, r(20).trim, r(21).trim.toInt, r(22).trim, r23, r(24).trim,
> r(25).trim, r26, r27, r(28).trim)*
> }).registerAsTable("sdp")
>
> Thanks
> Best Regards
>
> On Wed, Feb 25, 2015 at 2:14 PM, anamika gupta
> wrote:
>
>> The link has
sses.html>, here
> <http://www.scala-lang.org/docu/files/ScalaReference.pdf> and in Programming
> in Scala <http://www.artima.com/shop/programming_in_scala_2ed> ch 15.
>
> Hope that helps!
>
> Best,
>
> Patrick
>
> On Thu, Feb 26, 2015 at 6:37 AM, ana
I have three tables with the following schema:
case class* date_d*(WID: Int, CALENDAR_DATE: java.sql.Timestamp,
DATE_STRING: String, DAY_OF_WEEK: String, DAY_OF_MONTH: Int, DAY_OF_YEAR:
Int, END_OF_MONTH_FLAG: String, YEARWEEK: Int, CALENDAR_MONTH: String,
MONTH_NUM: Int, YEARMONTH: Int, QUARTER:
The issue is now resolved.
One of the csv files had an incorrect record at the end.
On Fri, Feb 27, 2015 at 4:24 PM, anamika gupta
wrote:
> I have three tables with the following schema:
>
> case class* date_d*(WID: Int, CALENDAR_DATE: java.sql.Timestamp,
> DATE_STRING: String,
Please fine the query plan
scala> sqlContext.sql("SELECT dw.DAY_OF_WEEK, dw.HOUR, avg(dw.SDP_USAGE) AS
AVG_SDP_USAGE FROM (SELECT sdp.WID, DAY_OF_WEEK, HOUR, SUM(INTERVAL_VALUE)
AS SDP_USAGE FROM (SELECT * FROM date_d AS dd JOIN interval_f AS intf ON
intf.DATE_WID = dd.WID WHERE intf.DATE_WID >= 2
Yeah thanks, I can now see the memory usage.
Please also verify if bytes read == Combined size of all RDDs ?
Actually, all my RDDs are completely cached in memory. So, Combined size of
my RDDs = Mem used (verified from WebUI)
On Fri, Mar 20, 2015 at 12:07 PM, Akhil Das
wrote:
> You could do a
