+1 (non-binding)
Tested Mesos coarse/fine-grained mode with 4 nodes Mesos cluster with
simple shuffle/map task.
Will be testing with more complete suite (ie: spark-perf) once the
infrastructure is setup to do so.
Tim
On Thu, Feb 19, 2015 at 12:50 PM, Krishna Sankar <ksanka...@gmail.com> wrote:
> Excellent. Explicit toDF() works.
> a) employees.toDF().registerTempTable("Employees") - works
> b) Also affects saveAsParquetFile - orders.toDF().saveAsParquetFile
>
> Adding to my earlier tests:
> 4.0 SQL from Scala and Python
> 4.1 result = sqlContext.sql("SELECT * from Employees WHERE State = 'WA'") OK
> 4.2 result = sqlContext.sql("SELECT
> OrderDetails.OrderID,ShipCountry,UnitPrice,Qty,Discount FROM Orders INNER
> JOIN OrderDetails ON Orders.OrderID = OrderDetails.OrderID") OK
> 4.3 result = sqlContext.sql("SELECT ShipCountry, Sum(OrderDetails.UnitPrice
> * Qty * Discount) AS ProductSales FROM Orders INNER JOIN OrderDetails ON
> Orders.OrderID = OrderDetails.OrderID GROUP BY ShipCountry") OK
> 4.4 saveAsParquetFile OK
> 4.5 Read and verify the 4.4 save - sqlContext.parquetFile,
> registerTempTable, sql OK
>
> Cheers & thanks Michael
> <k/>
>
>
>
> On Thu, Feb 19, 2015 at 12:02 PM, Michael Armbrust <mich...@databricks.com>
> wrote:
>
>> P.S: For some reason replacing "import sqlContext.createSchemaRDD" with "
>>> import sqlContext.implicits._" doesn't do the implicit conversations.
>>> registerTempTable
>>> gives syntax error. I will dig deeper tomorrow. Has anyone seen this ?
>>
>>
>> We will write up a whole migration guide before the final release, but I
>> can quickly explain this one. We made the implicit conversion
>> significantly less broad to avoid the chance of confusing conflicts.
>> However, now you have to call .toDF in order to force RDDs to become
>> DataFrames.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
