Not for sure, but I think it is bug as of 1.5.
Spark is using LIMIT keyword whether a table exists.
https://github.com/apache/spark/blob/branch-1.5/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L48
If your database does not support LIMIT keyword such as S
> wrote:
>
>> I would be great if you could try sql("SET
>> spark.sql.inMemoryColumnarStorage.partitionPruning=false") also, try Spark
>> 1.5.2-RC2
>> <http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/>
>>
>> On Fri, Nov 6, 2015 at 4:49 AM, Se
targetEntityId: String, properties:
> String)
> eventRDD.map{case (id, event, entityType, entityId, targetEntityType,
> targetEntityId,
> properties) =>
> Event(id, event, entityType, entityId, targetEntityType, targetEntityId,
> properties) }.toDF()
>
> The comparison === sho
Hi, Yin
Thanks for your time. This is the result.
--
scala> eventDF.filter($"entityType" ===
"user").select("entityId").distinct.explain(true)
== Parsed Logical Plan ==
Aggregate [entityId#16], [entityId#16]
Project [entityId#16]
Filter (entityType#15 = user)
Project [_1#0 A
