[jira] [Commented] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

2017-11-24 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265543#comment-16265543 ] Ryan Williams commented on SPARK-22393: --- Just confirming: it's expected that the Scala 2.11 repl

[jira] [Commented] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

2017-10-29 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224158#comment-16224158 ] Ryan Williams commented on SPARK-22393: --- Everything works fine in a Scala shell ({{scala -cp

[jira] [Updated] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

2017-10-29 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams updated SPARK-22393: -- Affects Version/s: (was: 2.0.0) 2.0.2 > spark-shell can't find

[jira] [Created] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

2017-10-29 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-22393: - Summary: spark-shell can't find imported types in class constructors, extends clause Key: SPARK-22393 URL: https://issues.apache.org/jira/browse/SPARK-22393

[jira] [Created] (SPARK-22328) ClosureCleaner misses referenced superclass fields, gives them null values

2017-10-21 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-22328: - Summary: ClosureCleaner misses referenced superclass fields, gives them null values Key: SPARK-22328 URL: https://issues.apache.org/jira/browse/SPARK-22328

[jira] [Commented] (SPARK-22288) Tricky interaction between closure-serialization and inheritance results in confusing failure

2017-10-17 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207706#comment-16207706 ] Ryan Williams commented on SPARK-22288: --- Makes sense, fine with me to "Won't Fix". bq. You can

[jira] [Created] (SPARK-22288) Tricky interaction between closure-serialization and inheritance results in confusing failure

2017-10-16 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-22288: - Summary: Tricky interaction between closure-serialization and inheritance results in confusing failure Key: SPARK-22288 URL: https://issues.apache.org/jira/browse/SPARK-22288

[jira] [Commented] (SPARK-21682) Caching 100k-task RDD GC-kills driver (due to updatedBlockStatuses?)

2017-08-09 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120537#comment-16120537 ] Ryan Williams commented on SPARK-21682: --- bq. But do you really need to create so many partitions?

[jira] [Updated] (SPARK-21682) Caching 100k-task RDD GC-kills driver (due to updatedBlockStatuses?)

2017-08-09 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams updated SPARK-21682: -- Description: h3. Summary * {{sc.parallelize(1 to 10, 10).cache.count}} causes a

[jira] [Comment Edited] (SPARK-21682) Caching 100k-task RDD GC-kills driver (due to updatedBlockStatuses?)

2017-08-09 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120409#comment-16120409 ] Ryan Williams edited comment on SPARK-21682 at 8/9/17 6:28 PM: ---

[jira] [Updated] (SPARK-21682) Caching 100k-task RDD GC-kills driver (due to updatedBlockStatuses?)

2017-08-09 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams updated SPARK-21682: -- Summary: Caching 100k-task RDD GC-kills driver (due to updatedBlockStatuses?) (was: Caching

[jira] [Updated] (SPARK-21682) Caching 100k-task RDD GC-kills driver due to updatedBlockStatuses

2017-08-09 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams updated SPARK-21682: -- Affects Version/s: 2.0.2 > Caching 100k-task RDD GC-kills driver due to updatedBlockStatuses >

[jira] [Updated] (SPARK-21682) Caching 100k-task RDD GC-kills driver (due to updatedBlockStatuses?)

2017-08-09 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams updated SPARK-21682: -- Description: h3. Summary * * {{internal.metrics.updatedBlockStatuses}} breaks a contract

[jira] [Commented] (SPARK-21682) Caching 100k-task RDD GC-kills driver due to updatedBlockStatuses

2017-08-09 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120409#comment-16120409 ] Ryan Williams commented on SPARK-21682: --- Interestingly, I thought the {{updatedBlockStatuses}}

[jira] [Created] (SPARK-21683) "TaskKilled (another attempt succeeded)" log message should be INFO level, not WARN

2017-08-09 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-21683: - Summary: "TaskKilled (another attempt succeeded)" log message should be INFO level, not WARN Key: SPARK-21683 URL: https://issues.apache.org/jira/browse/SPARK-21683

[jira] [Created] (SPARK-21682) Caching 100k-task RDD GC-kills driver due to updatedBlockStatuses

2017-08-09 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-21682: - Summary: Caching 100k-task RDD GC-kills driver due to updatedBlockStatuses Key: SPARK-21682 URL: https://issues.apache.org/jira/browse/SPARK-21682 Project: Spark

[jira] [Commented] (SPARK-21569) Internal Spark class needs to be kryo-registered

2017-07-30 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106530#comment-16106530 ] Ryan Williams commented on SPARK-21569: --- Main.scala: {code} package com.foo import

[jira] [Created] (SPARK-21569) Internal Spark class needs to be kryo-registered

2017-07-28 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-21569: - Summary: Internal Spark class needs to be kryo-registered Key: SPARK-21569 URL: https://issues.apache.org/jira/browse/SPARK-21569 Project: Spark Issue

[jira] [Comment Edited] (SPARK-16599) java.util.NoSuchElementException: None.get at at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask(BlockInfoManager.scala:343)

2017-07-22 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097318#comment-16097318 ] Ryan Williams edited comment on SPARK-16599 at 7/22/17 1:23 PM: Seems

[jira] [Commented] (SPARK-16599) java.util.NoSuchElementException: None.get at at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask(BlockInfoManager.scala:343)

2017-07-22 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097318#comment-16097318 ] Ryan Williams commented on SPARK-16599: --- Seems like the second {{SparkContext}}'s

[jira] [Commented] (SPARK-16599) java.util.NoSuchElementException: None.get at at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask(BlockInfoManager.scala:343)

2017-07-22 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097302#comment-16097302 ] Ryan Williams commented on SPARK-16599: --- Update: I think the real story here is the first stack

[jira] [Commented] (SPARK-16599) java.util.NoSuchElementException: None.get at at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask(BlockInfoManager.scala:343)

2017-07-21 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097070#comment-16097070 ] Ryan Williams commented on SPARK-16599: --- I've been running into this lately, and I just [made a

[jira] [Commented] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-20 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095678#comment-16095678 ] Ryan Williams commented on SPARK-21425: --- [~zsxwing] yea, it's static accumulators, and seems to

[jira] [Comment Edited] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-20 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090017#comment-16090017 ] Ryan Williams edited comment on SPARK-21425 at 7/21/17 2:12 AM: Yea,

[jira] [Commented] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-17 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090017#comment-16090017 ] Ryan Williams commented on SPARK-21425: --- Yea, static accumulators; I was able to reproduce it in

[jira] [Updated] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-16 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams updated SPARK-21425: -- Priority: Minor (was: Major) > LongAccumulator, DoubleAccumulator not threadsafe >

[jira] [Commented] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-16 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088919#comment-16088919 ] Ryan Williams commented on SPARK-21425: --- Digging a bit more, it seems like: - there's no special

[jira] [Comment Edited] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-16 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088909#comment-16088909 ] Ryan Williams edited comment on SPARK-21425 at 7/16/17 12:25 PM: -

[jira] [Commented] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-16 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088909#comment-16088909 ] Ryan Williams commented on SPARK-21425: --- [~sowen] interesting, I didn't think my example was using

[jira] [Created] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-15 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-21425: - Summary: LongAccumulator, DoubleAccumulator not threadsafe Key: SPARK-21425 URL: https://issues.apache.org/jira/browse/SPARK-21425 Project: Spark Issue

[jira] [Commented] (SPARK-21143) Fail to fetch blocks >1MB in size in presence of conflicting Netty version

2017-06-19 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054707#comment-16054707 ] Ryan Williams commented on SPARK-21143: --- [~zsxwing] bq. it's too risky to upgrade from 4.0.X to

[jira] [Created] (SPARK-21143) Fail to fetch blocks >1MB in size in presence of conflicting Netty version

2017-06-19 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-21143: - Summary: Fail to fetch blocks >1MB in size in presence of conflicting Netty version Key: SPARK-21143 URL: https://issues.apache.org/jira/browse/SPARK-21143

[jira] [Comment Edited] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2017-04-10 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963304#comment-15963304 ] Ryan Williams edited comment on SPARK-650 at 4/10/17 6:42 PM: -- Both suggested

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2017-04-10 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963304#comment-15963304 ] Ryan Williams commented on SPARK-650: - Both suggested workarounds here are lacking or broken / actively

[jira] [Commented] (SPARK-9039) Jobs page shows nonsensical task-progress-bar numbers when speculation occurs

2015-12-03 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038027#comment-15038027 ] Ryan Williams commented on SPARK-9039: -- Just saw this in Spark 1.5.2

[jira] [Commented] (SPARK-6497) Class is not registered: scala.reflect.ManifestFactory$$anon$9

2015-12-01 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035272#comment-15035272 ] Ryan Williams commented on SPARK-6497: -- I just saw this on 1.5.2:

[jira] [Updated] (SPARK-6497) Class is not registered: scala.reflect.ManifestFactory$$anon$9

2015-12-01 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams updated SPARK-6497: - Affects Version/s: 1.5.2 > Class is not registered: scala.reflect.ManifestFactory$$anon$9 >

[jira] [Commented] (SPARK-5784) Add StatsDSink to MetricsSystem

2015-11-21 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15020742#comment-15020742 ] Ryan Williams commented on SPARK-5784: -- sorry [~anubhavk], I haven't looked at this in a while and

[jira] [Commented] (SPARK-5784) Add StatsDSink to MetricsSystem

2015-11-17 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009681#comment-15009681 ] Ryan Williams commented on SPARK-5784: -- You'd have to build it yourself; feel free to use the code in

[jira] [Commented] (SPARK-9039) Jobs page shows nonsensical task-progress-bar numbers when speculation occurs

2015-11-09 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14997263#comment-14997263 ] Ryan Williams commented on SPARK-9039: -- I don't remember but will ping here if I see it again. >

[jira] [Commented] (SPARK-11459) Allow configuring checkpoint dir, filenames

2015-11-04 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990676#comment-14990676 ] Ryan Williams commented on SPARK-11459: --- I'm mostly interested in saving RDDs to disk with

[jira] [Created] (SPARK-11459) Allow configuring checkpoint dir, filenames

2015-11-02 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-11459: - Summary: Allow configuring checkpoint dir, filenames Key: SPARK-11459 URL: https://issues.apache.org/jira/browse/SPARK-11459 Project: Spark Issue Type:

[jira] [Created] (SPARK-11461) ObjectFile saving/loading should use configured serializer

2015-11-02 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-11461: - Summary: ObjectFile saving/loading should use configured serializer Key: SPARK-11461 URL: https://issues.apache.org/jira/browse/SPARK-11461 Project: Spark

[jira] [Commented] (SPARK-11162) Allow enabling debug logging from the command line

2015-10-23 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971520#comment-14971520 ] Ryan Williams commented on SPARK-11162: --- Makes sense, some googling has left me with the impression

[jira] [Commented] (SPARK-11162) Allow enabling debug logging from the command line

2015-10-23 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971361#comment-14971361 ] Ryan Williams commented on SPARK-11162: --- Do you know how I might enable debug logging with -D

[jira] [Updated] (SPARK-11285) Infinite TaskCommitDenied loop

2015-10-23 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams updated SPARK-11285: -- Description: I've seen several apps enter this failing state in the last couple of days. I've

[jira] [Commented] (SPARK-11285) Infinite TaskCommitDenied loop

2015-10-23 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971161#comment-14971161 ] Ryan Williams commented on SPARK-11285: --- A quick note on the structure of the linked directories

[jira] [Created] (SPARK-11285) Infinite TaskCommitDenied loop

2015-10-23 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-11285: - Summary: Infinite TaskCommitDenied loop Key: SPARK-11285 URL: https://issues.apache.org/jira/browse/SPARK-11285 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-11162) Allow enabling debug logging from the command line

2015-10-19 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963509#comment-14963509 ] Ryan Williams edited comment on SPARK-11162 at 10/19/15 4:01 PM: - In the

[jira] [Commented] (SPARK-11162) Allow enabling debug logging from the command line

2015-10-19 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963509#comment-14963509 ] Ryan Williams commented on SPARK-11162: --- In the second message (of 2 total, afaict) on the thread,

[jira] [Comment Edited] (SPARK-11162) Allow enabling debug logging from the command line

2015-10-19 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963509#comment-14963509 ] Ryan Williams edited comment on SPARK-11162 at 10/19/15 3:59 PM: - In the

[jira] [Commented] (SPARK-11161) Viewing the web UI for the first time unpersists a cached RDD

2015-10-19 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963556#comment-14963556 ] Ryan Williams commented on SPARK-11161: --- bq. I expect RDDs to behave like JVM objects in this

[jira] [Created] (SPARK-11156) Web UI doesn't count or show info about replicated blocks

2015-10-16 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-11156: - Summary: Web UI doesn't count or show info about replicated blocks Key: SPARK-11156 URL: https://issues.apache.org/jira/browse/SPARK-11156 Project: Spark

[jira] [Created] (SPARK-11162) Allow enabling debug logging from the command line

2015-10-16 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-11162: - Summary: Allow enabling debug logging from the command line Key: SPARK-11162 URL: https://issues.apache.org/jira/browse/SPARK-11162 Project: Spark Issue

[jira] [Commented] (SPARK-11161) Viewing the web UI for the first time unpersists a cached RDD

2015-10-16 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14961501#comment-14961501 ] Ryan Williams commented on SPARK-11161: --- Digging in to the code a bit, I determined that this

[jira] [Created] (SPARK-11161) Viewing the web UI for the first time unpersists a cached RDD

2015-10-16 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-11161: - Summary: Viewing the web UI for the first time unpersists a cached RDD Key: SPARK-11161 URL: https://issues.apache.org/jira/browse/SPARK-11161 Project: Spark

[jira] [Commented] (SPARK-11120) maxNumExecutorFailures defaults to 3 under dynamic allocation

2015-10-15 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14960169#comment-14960169 ] Ryan Williams commented on SPARK-11120: --- Without dynamic allocation, you are allowed [twice the

[jira] [Created] (SPARK-11120) maxNumExecutorFailures defaults to 3 under dynamic allocation

2015-10-14 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-11120: - Summary: maxNumExecutorFailures defaults to 3 under dynamic allocation Key: SPARK-11120 URL: https://issues.apache.org/jira/browse/SPARK-11120 Project: Spark

[jira] [Created] (SPARK-10871) Specify number of failed executors in ApplicationMaster error message

2015-09-29 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-10871: - Summary: Specify number of failed executors in ApplicationMaster error message Key: SPARK-10871 URL: https://issues.apache.org/jira/browse/SPARK-10871 Project:

[jira] [Created] (SPARK-10878) Race condition when resolving Maven coordinates via Ivy

2015-09-29 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-10878: - Summary: Race condition when resolving Maven coordinates via Ivy Key: SPARK-10878 URL: https://issues.apache.org/jira/browse/SPARK-10878 Project: Spark

[jira] [Created] (SPARK-10846) Stray META-INF in directory spark-shell is launched from causes problems

2015-09-27 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-10846: - Summary: Stray META-INF in directory spark-shell is launched from causes problems Key: SPARK-10846 URL: https://issues.apache.org/jira/browse/SPARK-10846 Project:

[jira] [Commented] (SPARK-7169) Allow to specify metrics configuration more flexibly

2015-09-14 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744200#comment-14744200 ] Ryan Williams commented on SPARK-7169: -- [~jlewandowski] I assume [~jerryshao] is referring to the

[jira] [Created] (SPARK-10551) Successful task-end event after task failed due to executor loss

2015-09-10 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-10551: - Summary: Successful task-end event after task failed due to executor loss Key: SPARK-10551 URL: https://issues.apache.org/jira/browse/SPARK-10551 Project: Spark

[jira] [Commented] (SPARK-10551) Successful task-end event after task failed due to executor loss

2015-09-10 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739615#comment-14739615 ] Ryan Williams commented on SPARK-10551: --- Here is the full event log:

[jira] [Commented] (SPARK-10551) Successful task-end event after task failed due to executor loss

2015-09-10 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739619#comment-14739619 ] Ryan Williams commented on SPARK-10551: --- The same behavior is observable on a second task,

[jira] [Updated] (SPARK-10551) Successful task-end event after task failed due to executor loss

2015-09-10 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams updated SPARK-10551: -- Description: Doing forensics on some failed Spark applications and seeing nonsensical things

[jira] [Commented] (SPARK-10551) Successful task-end event after task failed due to executor loss

2015-09-10 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739630#comment-14739630 ] Ryan Williams commented on SPARK-10551: --- Something else I just noticed: both of the

[jira] [Commented] (SPARK-1517) Publish nightly snapshots of documentation, maven artifacts, and binary builds

2015-08-06 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660816#comment-14660816 ] Ryan Williams commented on SPARK-1517: -- That all makes sense, thanks. I guess I was

[jira] [Commented] (SPARK-1517) Publish nightly snapshots of documentation, maven artifacts, and binary builds

2015-08-06 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660374#comment-14660374 ] Ryan Williams commented on SPARK-1517: -- Hey [~pwendell], thanks for continuing to

[jira] [Commented] (SPARK-1517) Publish nightly snapshots of documentation, maven artifacts, and binary builds

2015-08-06 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660553#comment-14660553 ] Ryan Williams commented on SPARK-1517: -- h3. Maven snapshots I hear your point that

[jira] [Commented] (SPARK-9608) Incorrect zinc -status check in build/mvn

2015-08-04 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14654235#comment-14654235 ] Ryan Williams commented on SPARK-9608: -- Sure, afk atm but will send one shortly

[jira] [Created] (SPARK-9607) Incorrect zinc check in build/mvn

2015-08-04 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-9607: Summary: Incorrect zinc check in build/mvn Key: SPARK-9607 URL: https://issues.apache.org/jira/browse/SPARK-9607 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-9607) Incorrect zinc check in build/mvn

2015-08-04 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14654249#comment-14654249 ] Ryan Williams commented on SPARK-9607: -- Not sure I have the right bash terminology,

[jira] [Created] (SPARK-9608) Incorrect zinc -status check in build/mvn

2015-08-04 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-9608: Summary: Incorrect zinc -status check in build/mvn Key: SPARK-9608 URL: https://issues.apache.org/jira/browse/SPARK-9608 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-9036) SparkListenerExecutorMetricsUpdate messages not included in JsonProtocol

2015-07-27 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643601#comment-14643601 ] Ryan Williams commented on SPARK-9036: -- Heh, [guilty as

[jira] [Created] (SPARK-9366) TaskEnd event emitted for task has different stage attempt ID than TaskStart for same task

2015-07-26 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-9366: Summary: TaskEnd event emitted for task has different stage attempt ID than TaskStart for same task Key: SPARK-9366 URL: https://issues.apache.org/jira/browse/SPARK-9366

[jira] [Commented] (SPARK-1301) Add UI elements to collapse Aggregated Metrics by Executor pane on stage page

2015-07-22 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14637299#comment-14637299 ] Ryan Williams commented on SPARK-1301: -- [~srowen] I read this as referring to the

[jira] [Commented] (SPARK-4024) Remember user preferences for metrics to show in the UI

2015-07-22 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14637306#comment-14637306 ] Ryan Williams commented on SPARK-4024: -- FWIW it seemed like [~zsxwing] solved some of

[jira] [Commented] (SPARK-9036) SparkListenerExecutorMetricsUpdate messages not included in JsonProtocol

2015-07-22 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14637768#comment-14637768 ] Ryan Williams commented on SPARK-9036: -- [~srowen] did you just assign this to me

[jira] [Created] (SPARK-9195) RDD/Storage metrics don't update cached partition counts when executors are removed/lost

2015-07-20 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-9195: Summary: RDD/Storage metrics don't update cached partition counts when executors are removed/lost Key: SPARK-9195 URL: https://issues.apache.org/jira/browse/SPARK-9195

[jira] [Commented] (SPARK-9197) Cached RDD partitions are lost when executors are dynamically deallocated

2015-07-20 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633755#comment-14633755 ] Ryan Williams commented on SPARK-9197: -- {quote}Yes, that must necessarily happen, or

[jira] [Updated] (SPARK-9197) Cached RDD partitions are lost when executors are dynamically deallocated

2015-07-20 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams updated SPARK-9197: - Priority: Minor (was: Major) Cached RDD partitions are lost when executors are dynamically

[jira] [Commented] (SPARK-9038) Missing TaskEnd event when task attempt is superseded by another (speculative) attempt

2015-07-20 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633723#comment-14633723 ] Ryan Williams commented on SPARK-9038: -- Note: this also causes Spark to think that

[jira] [Created] (SPARK-9197) Cached RDD partitions are lost when executors are dynamically deallocated

2015-07-20 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-9197: Summary: Cached RDD partitions are lost when executors are dynamically deallocated Key: SPARK-9197 URL: https://issues.apache.org/jira/browse/SPARK-9197 Project:

[jira] [Created] (SPARK-9036) SparkListenerExecutorMetricsUpdate messages not included in JsonProtocol

2015-07-14 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-9036: Summary: SparkListenerExecutorMetricsUpdate messages not included in JsonProtocol Key: SPARK-9036 URL: https://issues.apache.org/jira/browse/SPARK-9036 Project:

[jira] [Created] (SPARK-9038) Missing TaskEnd event when task attempt is superseded by another (speculative) attempt

2015-07-14 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-9038: Summary: Missing TaskEnd event when task attempt is superseded by another (speculative) attempt Key: SPARK-9038 URL: https://issues.apache.org/jira/browse/SPARK-9038

[jira] [Created] (SPARK-9039) Jobs page shows nonsensical task-progress-bar numbers when speculation occurs

2015-07-14 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-9039: Summary: Jobs page shows nonsensical task-progress-bar numbers when speculation occurs Key: SPARK-9039 URL: https://issues.apache.org/jira/browse/SPARK-9039 Project:

[jira] [Resolved] (SPARK-6447) Add quick-links to StagePage to jump to Accumulator/Task tables

2015-07-10 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams resolved SPARK-6447. -- Resolution: Duplicate Add quick-links to StagePage to jump to Accumulator/Task tables

[jira] [Commented] (SPARK-5784) Add StatsDSink to MetricsSystem

2015-06-06 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575788#comment-14575788 ] Ryan Williams commented on SPARK-5784: -- Hi [~varvind], the verdict here was that this

[jira] [Created] (SPARK-8088) ExecutionAllocationManager spamming INFO logs about Lowering target number of executors

2015-06-03 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-8088: Summary: ExecutionAllocationManager spamming INFO logs about Lowering target number of executors Key: SPARK-8088 URL: https://issues.apache.org/jira/browse/SPARK-8088

[jira] [Closed] (SPARK-5784) Add StatsDSink to MetricsSystem

2015-06-02 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams closed SPARK-5784. Resolution: Not A Problem Add StatsDSink to MetricsSystem ---

[jira] [Commented] (SPARK-5784) Add StatsDSink to MetricsSystem

2015-06-02 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569655#comment-14569655 ] Ryan Williams commented on SPARK-5784: -- [~varvind] seems like no; Spark packages

[jira] [Created] (SPARK-7901) Attempt to request negative number of executors with dynamic allocation

2015-05-27 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-7901: Summary: Attempt to request negative number of executors with dynamic allocation Key: SPARK-7901 URL: https://issues.apache.org/jira/browse/SPARK-7901 Project: Spark

[jira] [Commented] (SPARK-7901) Attempt to request negative number of executors with dynamic allocation

2015-05-27 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14561484#comment-14561484 ] Ryan Williams commented on SPARK-7901: -- Looks like a dupe of

[jira] [Resolved] (SPARK-7901) Attempt to request negative number of executors with dynamic allocation

2015-05-27 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams resolved SPARK-7901. -- Resolution: Duplicate Attempt to request negative number of executors with dynamic allocation

[jira] [Created] (SPARK-6543) Scheduler Delay column in tasks table on Stage detail page showing negative numbers

2015-03-25 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-6543: Summary: Scheduler Delay column in tasks table on Stage detail page showing negative numbers Key: SPARK-6543 URL: https://issues.apache.org/jira/browse/SPARK-6543

[jira] [Closed] (SPARK-6543) Scheduler Delay column in tasks table on Stage detail page showing negative numbers

2015-03-25 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams closed SPARK-6543. Resolution: Duplicate Fix Version/s: 1.3.0 Scheduler Delay column in tasks table on Stage

[jira] [Commented] (SPARK-6543) Scheduler Delay column in tasks table on Stage detail page showing negative numbers

2015-03-25 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381388#comment-14381388 ] Ryan Williams commented on SPARK-6543: -- ah, right you are, fixed in 1.3.0, thanks

[jira] [Created] (SPARK-6541) Executor table on Stage page should sort by Executor ID numerically, not lexically

2015-03-25 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-6541: Summary: Executor table on Stage page should sort by Executor ID numerically, not lexically Key: SPARK-6541 URL: https://issues.apache.org/jira/browse/SPARK-6541

[jira] [Commented] (SPARK-6449) Driver OOM results in reported application result SUCCESS

2015-03-23 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377249#comment-14377249 ] Ryan Williams commented on SPARK-6449: -- It doesn't look like it; [here is a

  1   2   >