[jira] [Commented] (SPARK-12837) Spark driver requires large memory space for serialized results even there are no data collected to the driver
[
https://issues.apache.org/jira/browse/SPARK-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125726#comment-16125726
]
Todd Leo commented on SPARK-12837:
--
[~cloud_fan] Is there a ticket tracking "driver-free broadcast" ?
> Spark driver requires large memory space for serialized results even there
> are no data collected to the driver
> --
>
> Key: SPARK-12837
> URL: https://issues.apache.org/jira/browse/SPARK-12837
> Project: Spark
> Issue Type: Question
> Components: SQL
>Affects Versions: 1.5.2, 1.6.0
>Reporter: Tien-Dung LE
>Assignee: Wenchen Fan
>Priority: Critical
> Fix For: 2.2.0
>
>
> Executing a sql statement with a large number of partitions requires a high
> memory space for the driver even there are no requests to collect data back
> to the driver.
> Here are steps to re-produce the issue.
> 1. Start spark shell with a spark.driver.maxResultSize setting
> {code:java}
> bin/spark-shell --driver-memory=1g --conf spark.driver.maxResultSize=1m
> {code}
> 2. Execute the code
> {code:java}
> case class Toto( a: Int, b: Int)
> val df = sc.parallelize( 1 to 1e6.toInt).map( i => Toto( i, i)).toDF
> sqlContext.setConf( "spark.sql.shuffle.partitions", "200" )
> df.groupBy("a").count().saveAsParquetFile( "toto1" ) // OK
> sqlContext.setConf( "spark.sql.shuffle.partitions", 1e3.toInt.toString )
> df.repartition(1e3.toInt).groupBy("a").count().repartition(1e3.toInt).saveAsParquetFile(
> "toto2" ) // ERROR
> {code}
> The error message is
> {code:java}
> Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
> Total size of serialized results of 393 tasks (1025.9 KB) is bigger than
> spark.driver.maxResultSize (1024.0 KB)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13298) DAG visualization does not render correctly for jobs
[
https://issues.apache.org/jira/browse/SPARK-13298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15192324#comment-15192324
]
Todd Leo commented on SPARK-13298:
--
Pls also see SPARK-13645: DAG Diagram not shown properly in Chrome
> DAG visualization does not render correctly for jobs
>
>
> Key: SPARK-13298
> URL: https://issues.apache.org/jira/browse/SPARK-13298
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 1.6.0
>Reporter: Lucas Woltmann
>Assignee: Shixiong Zhu
> Fix For: 1.6.1, 2.0.0
>
> Attachments: dag_full.png, dag_viz.png, no-dag.png
>
>
> Whenever I try to open the DAG for a job, I get something like this:
> !dag_viz.png!
> Obviously the svg doesn't get resized, but if I resize it manually, only the
> first of four stages in the DAG is shown.
> The js console says (variable v is null in peg$c34):
> {code:javascript}
> Uncaught TypeError: Cannot read property '3' of null
> peg$c34 @ graphlib-dot.min.js:1
> peg$parseidDef @ graphlib-dot.min.js:1
> peg$parseaList @ graphlib-dot.min.js:1
> peg$parseattrListBlock @ graphlib-dot.min.js:1
> peg$parseattrList @ graphlib-dot.min.js:1
> peg$parsenodeStmt @ graphlib-dot.min.js:1
> peg$parsestmt @ graphlib-dot.min.js:1
> peg$parsestmtList @ graphlib-dot.min.js:1
> peg$parsesubgraphStmt @ graphlib-dot.min.js:1
> peg$parsenodeIdOrSubgraph @ graphlib-dot.min.js:1
> peg$parseedgeStmt @ graphlib-dot.min.js:1
> peg$parsestmt @ graphlib-dot.min.js:1
> peg$parsestmtList @ graphlib-dot.min.js:1
> peg$parsesubgraphStmt @ graphlib-dot.min.js:1
> peg$parsenodeIdOrSubgraph @ graphlib-dot.min.js:1
> peg$parseedgeStmt @ graphlib-dot.min.js:1
> peg$parsestmt @ graphlib-dot.min.js:1
> peg$parsestmtList @ graphlib-dot.min.js:1
> peg$parsegraphStmt @ graphlib-dot.min.js:1
> parse @ graphlib-dot.min.js:2
> readOne @ graphlib-dot.min.js:2
> renderDot @ spark-dag-viz.js:281
> (anonymous function) @ spark-dag-viz.js:248
> (anonymous function) @ d3.min.js:
> 3Y @ d3.min.js:1
> _a.each @ d3.min.js:3
> renderDagVizForJob @ spark-dag-viz.js:207
> renderDagViz @ spark-dag-viz.js:163
> toggleDagViz @ spark-dag-viz.js:100
> onclick @ ?id=2:153
> {code}
> (tested in FIrefox 44.0.1 and Chromium 48.0.2564.103)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-13645) DAG Diagram not shown properly in Chrome
[ https://issues.apache.org/jira/browse/SPARK-13645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Leo updated SPARK-13645: - Description: In my Chrome 49, the execution DAG diagram can't be shown properly. Only a few grey dots lays there. Thought this is what I'm supposed to see at first. It works fine in Firefox, though. See the attachment below. was: In my Chrome 49, the execution DAG diagram can't be shown properly. Only a few grey dots lays there. Thought this is what I'm supposed to see at first. See the attachment below. > DAG Diagram not shown properly in Chrome > > > Key: SPARK-13645 > URL: https://issues.apache.org/jira/browse/SPARK-13645 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 1.6.0 > Environment: Chrome 49, 64-bit >Reporter: Todd Leo > Attachments: Slack for iOS Upload.jpg > > > In my Chrome 49, the execution DAG diagram can't be shown properly. Only a > few grey dots lays there. Thought this is what I'm supposed to see at first. > It works fine in Firefox, though. > See the attachment below. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-13645) DAG Diagram not shown properly in Chrome
[ https://issues.apache.org/jira/browse/SPARK-13645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Leo updated SPARK-13645: - Description: In my Chrome 49, the execution DAG diagram can't be shown properly. Only a few grey dots lays there. Thought this is what I'm supposed to see at first. See the attachment below. was:In my Chrome 49, the execution DAG diagram can't be shown properly. Only a few grey dots lays there. Thought this is what I'm supposed to see at first. > DAG Diagram not shown properly in Chrome > > > Key: SPARK-13645 > URL: https://issues.apache.org/jira/browse/SPARK-13645 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 1.6.0 > Environment: Chrome 49, 64-bit >Reporter: Todd Leo > Attachments: Slack for iOS Upload.jpg > > > In my Chrome 49, the execution DAG diagram can't be shown properly. Only a > few grey dots lays there. Thought this is what I'm supposed to see at first. > See the attachment below. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-13645) DAG Diagram not shown properly in Chrome
[ https://issues.apache.org/jira/browse/SPARK-13645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Leo updated SPARK-13645: - Attachment: Slack for iOS Upload.jpg > DAG Diagram not shown properly in Chrome > > > Key: SPARK-13645 > URL: https://issues.apache.org/jira/browse/SPARK-13645 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 1.6.0 > Environment: Chrome 49, 64-bit >Reporter: Todd Leo > Attachments: Slack for iOS Upload.jpg > > > In my Chrome 49, the execution DAG diagram can't be shown properly. Only a > few grey dots lays there. Thought this is what I'm supposed to see at first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-13645) DAG Diagram not shown properly in Chrome
Todd Leo created SPARK-13645: Summary: DAG Diagram not shown properly in Chrome Key: SPARK-13645 URL: https://issues.apache.org/jira/browse/SPARK-13645 Project: Spark Issue Type: Bug Components: Web UI Affects Versions: 1.6.0 Environment: Chrome 49, 64-bit Reporter: Todd Leo In my Chrome 49, the execution DAG diagram can't be shown properly. Only a few grey dots lays there. Thought this is what I'm supposed to see at first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
