+1 (non-binding) Tested Mesos coarse/fine-grained mode with 4 nodes Mesos cluster with simple shuffle/map task.
Will be testing with more complete suite (ie: spark-perf) once the infrastructure is setup to do so. Tim On Thu, Feb 19, 2015 at 12:50 PM, Krishna Sankar <[email protected]> wrote: > Excellent. Explicit toDF() works. > a) employees.toDF().registerTempTable("Employees") - works > b) Also affects saveAsParquetFile - orders.toDF().saveAsParquetFile > > Adding to my earlier tests: > 4.0 SQL from Scala and Python > 4.1 result = sqlContext.sql("SELECT * from Employees WHERE State = 'WA'") OK > 4.2 result = sqlContext.sql("SELECT > OrderDetails.OrderID,ShipCountry,UnitPrice,Qty,Discount FROM Orders INNER > JOIN OrderDetails ON Orders.OrderID = OrderDetails.OrderID") OK > 4.3 result = sqlContext.sql("SELECT ShipCountry, Sum(OrderDetails.UnitPrice > * Qty * Discount) AS ProductSales FROM Orders INNER JOIN OrderDetails ON > Orders.OrderID = OrderDetails.OrderID GROUP BY ShipCountry") OK > 4.4 saveAsParquetFile OK > 4.5 Read and verify the 4.4 save - sqlContext.parquetFile, > registerTempTable, sql OK > > Cheers & thanks Michael > <k/> > > > > On Thu, Feb 19, 2015 at 12:02 PM, Michael Armbrust <[email protected]> > wrote: > >> P.S: For some reason replacing "import sqlContext.createSchemaRDD" with " >>> import sqlContext.implicits._" doesn't do the implicit conversations. >>> registerTempTable >>> gives syntax error. I will dig deeper tomorrow. Has anyone seen this ? >> >> >> We will write up a whole migration guide before the final release, but I >> can quickly explain this one. We made the implicit conversion >> significantly less broad to avoid the chance of confusing conflicts. >> However, now you have to call .toDF in order to force RDDs to become >> DataFrames. >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
