Also bear in mind that explain() method call works on transformations
(Transformations are just manipulations of the data.).
examples filter, map, orderBy etc
scala> var y =
HiveContext.table("sales").select("time_id").agg(max("time_id")).explain(true)
== Parsed Logical Plan ==
'Aggregate [max('time_id) AS max(time_id)#359]
Project [time_id#354]
MetastoreRelation oraclehadoop, sales, None
== Analyzed Logical Plan ==
max(time_id): timestamp
Aggregate [max(time_id#354) AS max(time_id)#359]
Project [time_id#354]
MetastoreRelation oraclehadoop, sales, None
== Optimized Logical Plan ==
Aggregate [max(time_id#354) AS max(time_id)#359]
Project [time_id#354]
MetastoreRelation oraclehadoop, sales, None
== Physical Plan ==
TungstenAggregate(key=[],
functions=[(max(time_id#354),mode=Final,isDistinct=false)],
output=[max(time_id)#359])
TungstenExchange SinglePartition
TungstenAggregate(key=[],
functions=[(max(time_id#354),mode=Partial,isDistinct=false)],
output=[max#363])
HiveTableScan [time_id#354], (MetastoreRelation oraclehadoop, sales,
None)
Code Generation: true
y: Unit = ()
On 24/02/2016 09:49, Ashok Kumar wrote:
> looks useful thanks
>
> On Wednesday, 24 February 2016, 9:42, Yin Yang <yy201...@gmail.com> wrote:
>
> Is the following what you were looking for ?
>
> sqlContext.sql("""
> CREATE TEMPORARY TABLE partitionedParquet
> USING org.apache.spark.sql.parquet
> OPTIONS (
> path '/tmp/partitioned'
> )""")
>
> table("partitionedParquet").explain(true)
>
> On Wed, Feb 24, 2016 at 1:16 AM, Ashok Kumar <ashok34...@yahoo.com.invalid>
> wrote:
>
>> Gurus,
>>
>> Is there anything like explain in Spark to see the execution plan in
>> functional programming?
>>
>> warm regards
--
Dr Mich Talebzadeh
LinkedIn
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
NOTE: The information in this email is proprietary and confidential.
This message is for the designated recipient only, if you are not the
intended recipient, you should destroy it immediately. Any information
in this message shall not be understood as given or endorsed by Cloud
Technology Partners Ltd, its subsidiaries or their employees, unless
expressly so stated. It is the responsibility of the recipient to ensure
that this email is virus free, therefore neither Cloud Technology
partners Ltd, its subsidiaries nor their employees accept any
responsibility.
