[
https://issues.apache.org/jira/browse/SPARK-14946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266327#comment-15266327
]
Raymond Honderdors commented on SPARK-14946:
--------------------------------------------
version 1.6.1 query plan
== Parsed Logical Plan ==
'Project [unresolvedalias(*)]
+- 'Join Inner, Some(('sd.campaignid = 'c.campaign_id))
:- 'UnresolvedRelation `pe_servingdata`, Some(sd)
+- 'UnresolvedRelation `pe_campaigns_gzip`, Some(c)
== Analyzed Logical Plan ==
originaltime: string, pluid: string, sdg: string, type: bigint, useragent:
string, utctime: string, diorigin: string, dbid: string, timeid: string,
browser: string, brandid: bigint, time: string, zip: string, dma: string,
ad_id: int, ismobile: string, privacy: string, df: string, userip: string,
agencyid: bigint, ta: string, mb: string, advertiserid: bigint, campaignid:
bigint, os: string, usr: string, isdefaultimg: string, isuserinit: string,
impressiontype: string, referrer: string, city: string, masteradid: bigint,
state: string, val: string, isclick: string, flightid: bigint, siteid: string,
intrn: string, asset: string, sid: string, account_id: bigint, event_time:
bigint, campaign_id: bigint, campaign_type_id: int, campaign_name: string,
version: int, account_id: bigint
Project
[originaltime#2,pluid#3,sdg#4,type#5L,useragent#6,utctime#7,diorigin#8,dbid#9,timeid#10,browser#11,brandid#12L,time#13,zip#14,dma#15,ad_id#16,ismobile#17,privacy#18,df#19,userip#20,agencyid#21L,ta#22,mb#23,advertiserid#24L,campaignid#25L,os#26,usr#27,isdefaultimg#28,isuserinit#29,impressiontype#30,referrer#31,city#32,masteradid#33L,state#34,val#35,isclick#36,flightid#37L,siteid#38,intrn#39,asset#40,sid#41,account_id#0L,event_time#1L,campaign_id#43L,campaign_type_id#44,campaign_name#45,version#46,account_id#42L]
+- Join Inner, Some((campaignid#25L = campaign_id#43L))
:- Subquery sd
: +-
Relation[originaltime#2,pluid#3,sdg#4,type#5L,useragent#6,utctime#7,diorigin#8,dbid#9,timeid#10,browser#11,brandid#12L,time#13,zip#14,dma#15,ad_id#16,ismobile#17,privacy#18,df#19,userip#20,agencyid#21L,ta#22,mb#23,advertiserid#24L,campaignid#25L,os#26,usr#27,isdefaultimg#28,isuserinit#29,impressiontype#30,referrer#31,city#32,masteradid#33L,state#34,val#35,isclick#36,flightid#37L,siteid#38,intrn#39,asset#40,sid#41,account_id#0L,event_time#1L]
ParquetRelation: default.pe_servingdata
+- Subquery c
+-
Relation[campaign_id#43L,campaign_type_id#44,campaign_name#45,version#46,account_id#42L]
ParquetRelation: default.pe_campaigns_gzip
== Optimized Logical Plan ==
Project
[originaltime#2,pluid#3,sdg#4,type#5L,useragent#6,utctime#7,diorigin#8,dbid#9,timeid#10,browser#11,brandid#12L,time#13,zip#14,dma#15,ad_id#16,ismobile#17,privacy#18,df#19,userip#20,agencyid#21L,ta#22,mb#23,advertiserid#24L,campaignid#25L,os#26,usr#27,isdefaultimg#28,isuserinit#29,impressiontype#30,referrer#31,city#32,masteradid#33L,state#34,val#35,isclick#36,flightid#37L,siteid#38,intrn#39,asset#40,sid#41,account_id#0L,event_time#1L,campaign_id#43L,campaign_type_id#44,campaign_name#45,version#46,account_id#42L]
+- Join Inner, Some((campaignid#25L = campaign_id#43L))
:-
Relation[originaltime#2,pluid#3,sdg#4,type#5L,useragent#6,utctime#7,diorigin#8,dbid#9,timeid#10,browser#11,brandid#12L,time#13,zip#14,dma#15,ad_id#16,ismobile#17,privacy#18,df#19,userip#20,agencyid#21L,ta#22,mb#23,advertiserid#24L,campaignid#25L,os#26,usr#27,isdefaultimg#28,isuserinit#29,impressiontype#30,referrer#31,city#32,masteradid#33L,state#34,val#35,isclick#36,flightid#37L,siteid#38,intrn#39,asset#40,sid#41,account_id#0L,event_time#1L]
ParquetRelation: default.pe_servingdata
+-
Relation[campaign_id#43L,campaign_type_id#44,campaign_name#45,version#46,account_id#42L]
ParquetRelation: default.pe_campaigns_gzip
== Physical Plan ==
Project
[originaltime#2,pluid#3,sdg#4,type#5L,useragent#6,utctime#7,diorigin#8,dbid#9,timeid#10,browser#11,brandid#12L,time#13,zip#14,dma#15,ad_id#16,ismobile#17,privacy#18,df#19,userip#20,agencyid#21L,ta#22,mb#23,advertiserid#24L,campaignid#25L,os#26,usr#27,isdefaultimg#28,isuserinit#29,impressiontype#30,referrer#31,city#32,masteradid#33L,state#34,val#35,isclick#36,flightid#37L,siteid#38,intrn#39,asset#40,sid#41,account_id#0L,event_time#1L,campaign_id#43L,campaign_type_id#44,campaign_name#45,version#46,account_id#42L]
+- BroadcastHashJoin [campaignid#25L], [campaign_id#43L], BuildRight
:- Scan ParquetRelation:
default.pe_servingdata[originaltime#2,pluid#3,sdg#4,type#5L,useragent#6,utctime#7,diorigin#8,dbid#9,timeid#10,browser#11,brandid#12L,time#13,zip#14,dma#15,ad_id#16,ismobile#17,privacy#18,df#19,userip#20,agencyid#21L,ta#22,mb#23,advertiserid#24L,campaignid#25L,os#26,usr#27,isdefaultimg#28,isuserinit#29,impressiontype#30,referrer#31,city#32,masteradid#33L,state#34,val#35,isclick#36,flightid#37L,siteid#38,intrn#39,asset#40,sid#41,account_id#0L,event_time#1L]
InputPaths:
file:/home/raymond.honderdors/factdata/servingdata.parquet/account_id=15229/event_time=20160301
+- Scan ParquetRelation:
default.pe_campaigns_gzip[campaign_id#43L,campaign_type_id#44,campaign_name#45,version#46,account_id#42L]
InputPaths:
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=10272,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=10280,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=10532,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=11024,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=1144,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=11509,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=11774,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=12131,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=12185,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=12269,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=13246,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=13570,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=1358,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=13650,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=13680,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=1376,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=13856,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=14043,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=14087,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=1414,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=14154,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=14491,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=14938,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=15229,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=1553,
file:/home/raymond.honderdors/entities/p-campaigns-gzip.parquet/account_id=15678,
> Spark 2.0 vs 1.6.1 Query Time(out)
> ----------------------------------
>
> Key: SPARK-14946
> URL: https://issues.apache.org/jira/browse/SPARK-14946
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Raymond Honderdors
> Priority: Critical
> Attachments: Query Plan 1.6.1.png, screenshot-spark_2.0.png,
> spark-defaults.conf, spark-env.sh
>
>
> I run a query using JDBC driver running it on version 1.6.1 it return after 5
> – 6 min , the same query against version 2.0 fails after 2h (due to timeout)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
