[jira] [Commented] (SPARK-4017) Progress bar in console
[ https://issues.apache.org/jira/browse/SPARK-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285337#comment-14285337 ] Paul Wolfe commented on SPARK-4017: --- Nice, thanks. We are on logging level INFO, in my IDE it looks like its coming in stderr. Just to make sure were talking about the same thing, the bar looks like [Stage 7375:=(1 + 1) / 2][Stage 7398:=(1 + 1) / 2][Stage 7472:=(1 + 1) / 2] Progress bar in console --- Key: SPARK-4017 URL: https://issues.apache.org/jira/browse/SPARK-4017 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Davies Liu Assignee: Davies Liu Fix For: 1.2.0 It will be nice to have a progress bar in console, then we could change the default logging level to WARN. The progress bar should be in one line, could also be in the title of terminal. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4017) Progress bar in console
[ https://issues.apache.org/jira/browse/SPARK-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283848#comment-14283848 ] Paul Wolfe commented on SPARK-4017: --- Hello, was wondering if there is a way to turn this feature off? Clutters log files in java spark applications. Progress bar in console --- Key: SPARK-4017 URL: https://issues.apache.org/jira/browse/SPARK-4017 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Davies Liu Assignee: Davies Liu Fix For: 1.2.0 It will be nice to have a progress bar in console, then we could change the default logging level to WARN. The progress bar should be in one line, could also be in the title of terminal. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2316) StorageStatusListener should avoid O(blocks) operations
[
https://issues.apache.org/jira/browse/SPARK-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14269243#comment-14269243
]
Paul Wolfe commented on SPARK-2316:
---
Any workaround ideas for users who can't yet upgrade (stuck on version 1.0.0)?
StorageStatusListener should avoid O(blocks) operations
---
Key: SPARK-2316
URL: https://issues.apache.org/jira/browse/SPARK-2316
Project: Spark
Issue Type: Bug
Components: Spark Core, Web UI
Affects Versions: 1.0.0
Reporter: Patrick Wendell
Assignee: Andrew Or
Priority: Critical
Fix For: 1.1.0
In the case where jobs are frequently causing dropped blocks the storage
status listener can bottleneck. This is slow for a few reasons, one being
that we use Scala collection operations, the other being that we operations
that are O(number of blocks). I think using a few indices here could make
this much faster.
{code}
at java.lang.Integer.valueOf(Integer.java:642)
at scala.runtime.BoxesRunTime.boxToInteger(BoxesRunTime.java:70)
at
org.apache.spark.storage.StorageUtils$$anonfun$9.apply(StorageUtils.scala:82)
at
scala.collection.TraversableLike$$anonfun$groupBy$1.apply(TraversableLike.scala:328)
at
scala.collection.TraversableLike$$anonfun$groupBy$1.apply(TraversableLike.scala:327)
at
scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
at
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
at
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
at
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
at
scala.collection.TraversableLike$class.groupBy(TraversableLike.scala:327)
at scala.collection.AbstractTraversable.groupBy(Traversable.scala:105)
at
org.apache.spark.storage.StorageUtils$.rddInfoFromStorageStatus(StorageUtils.scala:82)
at
org.apache.spark.ui.storage.StorageListener.updateRDDInfo(StorageTab.scala:56)
at
org.apache.spark.ui.storage.StorageListener.onTaskEnd(StorageTab.scala:67)
- locked 0xa27ebe30 (a
org.apache.spark.ui.storage.StorageListener)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
