[jira] [Commented] (SPARK-1779) Warning when spark.storage.memoryFraction is not between 0 and 1

[ https://issues.apache.org/jira/browse/SPARK-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993901#comment-13993901 ] Erik Erlandson commented on SPARK-1779: --- I'll volunteer to take this, can somebody

[jira] [Commented] (SPARK-1781) Generalized validity checking for configuration parameters

[ https://issues.apache.org/jira/browse/SPARK-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993903#comment-13993903 ] Erik Erlandson commented on SPARK-1781: --- Ideally, pre-fab predicates could be

[jira] [Commented] (SPARK-1493) Apache RAT excludes don't work with file path (instead of file name)

[ https://issues.apache.org/jira/browse/SPARK-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025895#comment-14025895 ] Erik Erlandson commented on SPARK-1493: --- RAT itself appears to preclude exclusion

[jira] [Comment Edited] (SPARK-1493) Apache RAT excludes don't work with file path (instead of file name)

[ https://issues.apache.org/jira/browse/SPARK-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025895#comment-14025895 ] Erik Erlandson edited comment on SPARK-1493 at 6/9/14 11:13 PM:

[jira] [Commented] (SPARK-1493) Apache RAT excludes don't work with file path (instead of file name)

[ https://issues.apache.org/jira/browse/SPARK-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038921#comment-14038921 ] Erik Erlandson commented on SPARK-1493: --- I submitted a proposal patch for RAT-161,

[jira] [Commented] (SPARK-1856) Standardize MLlib interfaces

[ https://issues.apache.org/jira/browse/SPARK-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039495#comment-14039495 ] Erik Erlandson commented on SPARK-1856: --- How does this jira relate (if at all) to

[jira] [Commented] (SPARK-2262) Extreme Learning Machines (ELM) for MLLib

[ https://issues.apache.org/jira/browse/SPARK-2262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14042731#comment-14042731 ] Erik Erlandson commented on SPARK-2262: --- I'd like to have this assigned to me

[jira] [Created] (SPARK-2315) drop, dropRight and dropWhile which take RDD input and return RDD

Erik Erlandson created SPARK-2315: - Summary: drop, dropRight and dropWhile which take RDD input and return RDD Key: SPARK-2315 URL: https://issues.apache.org/jira/browse/SPARK-2315 Project: Spark

[jira] [Commented] (SPARK-2315) drop, dropRight and dropWhile which take RDD input and return RDD

[ https://issues.apache.org/jira/browse/SPARK-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046725#comment-14046725 ] Erik Erlandson commented on SPARK-2315: --- PR:

[jira] [Commented] (SPARK-1486) Support multi-model training in MLlib

[ https://issues.apache.org/jira/browse/SPARK-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062339#comment-14062339 ] Erik Erlandson commented on SPARK-1486: --- Does the dev on this issue effectively

[jira] [Commented] (SPARK-2488) Model SerDe in MLlib

[ https://issues.apache.org/jira/browse/SPARK-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062486#comment-14062486 ] Erik Erlandson commented on SPARK-2488: --- Related, possible duplicate: SPARK-1406

[jira] [Commented] (SPARK-2315) drop, dropRight and dropWhile which take RDD input and return RDD

[ https://issues.apache.org/jira/browse/SPARK-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079300#comment-14079300 ] Erik Erlandson commented on SPARK-2315: --- Updated the PR with a proper lazy-transform

[jira] [Commented] (SPARK-1021) sortByKey() launches a cluster job when it shouldn't

[ https://issues.apache.org/jira/browse/SPARK-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080250#comment-14080250 ] Erik Erlandson commented on SPARK-1021: --- I deferred the compute of the partition

[jira] [Commented] (SPARK-2911) provide rdd.parent[T](j) to obtain jth parent of rdd

[ https://issues.apache.org/jira/browse/SPARK-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090734#comment-14090734 ] Erik Erlandson commented on SPARK-2911: --- OK, shall I do it as part of this jira or

[jira] [Created] (SPARK-2991) RDD transforms for scan and scanLeft

Erik Erlandson created SPARK-2991: - Summary: RDD transforms for scan and scanLeft Key: SPARK-2991 URL: https://issues.apache.org/jira/browse/SPARK-2991 Project: Spark Issue Type: New

[jira] [Created] (SPARK-2992) The transforms formerly known as non-lazy

Erik Erlandson created SPARK-2992: - Summary: The transforms formerly known as non-lazy Key: SPARK-2992 URL: https://issues.apache.org/jira/browse/SPARK-2992 Project: Spark Issue Type:

[jira] [Updated] (SPARK-2991) RDD transforms for scan and scanLeft

[ https://issues.apache.org/jira/browse/SPARK-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-2991: -- Issue Type: Sub-task (was: New Feature) Parent: SPARK-2992 RDD transforms for scan

[jira] [Updated] (SPARK-1021) sortByKey() launches a cluster job when it shouldn't

[ https://issues.apache.org/jira/browse/SPARK-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-1021: -- Issue Type: Sub-task (was: Bug) Parent: SPARK-2992 sortByKey() launches a cluster

[jira] [Updated] (SPARK-2315) drop, dropRight and dropWhile which take RDD input and return RDD

[ https://issues.apache.org/jira/browse/SPARK-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-2315: -- Issue Type: Sub-task (was: New Feature) Parent: SPARK-2992 drop, dropRight and

[jira] [Created] (SPARK-3104) Jenkins failing to test some PRs when asked to

Erik Erlandson created SPARK-3104: - Summary: Jenkins failing to test some PRs when asked to Key: SPARK-3104 URL: https://issues.apache.org/jira/browse/SPARK-3104 Project: Spark Issue Type:

[jira] [Commented] (SPARK-2360) CSV import to SchemaRDDs

[ https://issues.apache.org/jira/browse/SPARK-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107233#comment-14107233 ] Erik Erlandson commented on SPARK-2360: --- It appears that this is not a pure lazy

[jira] [Commented] (SPARK-3250) More Efficient Sampling

[ https://issues.apache.org/jira/browse/SPARK-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114789#comment-14114789 ] Erik Erlandson commented on SPARK-3250: --- I did some experiments with sampling that

[jira] [Commented] (SPARK-3250) More Efficient Sampling

[ https://issues.apache.org/jira/browse/SPARK-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130809#comment-14130809 ] Erik Erlandson commented on SPARK-3250: --- I developed prototype iterator classes for

[jira] [Commented] (SPARK-3250) More Efficient Sampling

[ https://issues.apache.org/jira/browse/SPARK-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139965#comment-14139965 ] Erik Erlandson commented on SPARK-3250: --- PR:

[jira] [Created] (SPARK-4514) ComplexFutureAction does not preserve job group IDs

Erik Erlandson created SPARK-4514: - Summary: ComplexFutureAction does not preserve job group IDs Key: SPARK-4514 URL: https://issues.apache.org/jira/browse/SPARK-4514 Project: Spark Issue

[jira] [Commented] (SPARK-4514) ComplexFutureAction does not preserve job group IDs

[ https://issues.apache.org/jira/browse/SPARK-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14220439#comment-14220439 ] Erik Erlandson commented on SPARK-4514: --- Yes, I can cook up a PR. I've been

[jira] [Commented] (SPARK-4514) ComplexFutureAction does not preserve job group IDs

[ https://issues.apache.org/jira/browse/SPARK-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244592#comment-14244592 ] Erik Erlandson commented on SPARK-4514: --- The root problem appears to be the behavior

[jira] [Updated] (SPARK-4514) SparkContext localProperties does not inherit property updates across thread reuse

[ https://issues.apache.org/jira/browse/SPARK-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-4514: -- Summary: SparkContext localProperties does not inherit property updates across thread reuse

[jira] [Created] (SPARK-5876) generalize the type of categoricalFeaturesInfo to PartialFunction[Int, Int]

Erik Erlandson created SPARK-5876: - Summary: generalize the type of categoricalFeaturesInfo to PartialFunction[Int, Int] Key: SPARK-5876 URL: https://issues.apache.org/jira/browse/SPARK-5876 Project:

[jira] [Closed] (SPARK-2315) drop, dropRight and dropWhile which take RDD input and return RDD

[ https://issues.apache.org/jira/browse/SPARK-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson closed SPARK-2315. - Resolution: Won't Fix There is now an implementation on the silex project:

[jira] [Commented] (SPARK-2315) drop, dropRight and dropWhile which take RDD input and return RDD

[ https://issues.apache.org/jira/browse/SPARK-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14567478#comment-14567478 ] Erik Erlandson commented on SPARK-2315: --- The 'drop' RDD methods have been made

[jira] [Commented] (SPARK-2991) RDD transforms for scan and scanLeft

[ https://issues.apache.org/jira/browse/SPARK-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715417#comment-14715417 ] Erik Erlandson commented on SPARK-2991: --- This RFE now has a

[jira] [Issue Comment Deleted] (SPARK-15699) Add chi-squared test statistic as a split quality metric for decision trees

[ https://issues.apache.org/jira/browse/SPARK-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-15699: --- Comment: was deleted (was: Proposed PR: https://github.com/apache/spark/pull/13440) > Add

[jira] [Created] (SPARK-15699) Add chi-squared test statistic as a split quality metric for decision trees

Erik Erlandson created SPARK-15699: -- Summary: Add chi-squared test statistic as a split quality metric for decision trees Key: SPARK-15699 URL: https://issues.apache.org/jira/browse/SPARK-15699

[jira] [Commented] (SPARK-15699) Add chi-squared test statistic as a split quality metric for decision trees

[ https://issues.apache.org/jira/browse/SPARK-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310416#comment-15310416 ] Erik Erlandson commented on SPARK-15699: Proposed PR: https://github.com/apache/spark/pull/13438/

[jira] [Comment Edited] (SPARK-15699) Add chi-squared test statistic as a split quality metric for decision trees

[ https://issues.apache.org/jira/browse/SPARK-15699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310416#comment-15310416 ] Erik Erlandson edited comment on SPARK-15699 at 6/1/16 3:38 PM: Proposed

[jira] [Commented] (SPARK-7007) Add metrics source for ExecutorAllocationManager to expose internal status

[ https://issues.apache.org/jira/browse/SPARK-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437325#comment-15437325 ] Erik Erlandson commented on SPARK-7007: --- Are there instructions for how to enable these metrics?

[jira] [Comment Edited] (SPARK-7007) Add metrics source for ExecutorAllocationManager to expose internal status

[ https://issues.apache.org/jira/browse/SPARK-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437325#comment-15437325 ] Erik Erlandson edited comment on SPARK-7007 at 8/25/16 11:25 PM: - Are

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644841#comment-15644841 ] Erik Erlandson commented on SPARK-18278: I agree with [~willbenton] that Kube is a

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644919#comment-15644919 ] Erik Erlandson commented on SPARK-18278: Another comment on external plug-ins for scheduling:

[jira] [Created] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

Erik Erlandson created SPARK-18278: -- Summary: Support native submission of spark jobs to a kubernetes cluster Key: SPARK-18278 URL: https://issues.apache.org/jira/browse/SPARK-18278 Project: Spark

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637591#comment-15637591 ] Erik Erlandson commented on SPARK-18278: Current prototype:

[jira] [Updated] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-18278: --- External issue URL: https://github.com/kubernetes/kubernetes/issues/34377 External issue

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733439#comment-15733439 ] Erik Erlandson commented on SPARK-18278: As I understand it (and as I've built them) an "MVP"

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15718240#comment-15718240 ] Erik Erlandson commented on SPARK-18278: A possible scheme might be to publish the docker-files,

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15718249#comment-15718249 ] Erik Erlandson commented on SPARK-18278: Not publishing images puts users in the position of not

[jira] [Created] (SPARK-21277) Spark is invoking an incorrect serializer after UDAF completion

Erik Erlandson created SPARK-21277: -- Summary: Spark is invoking an incorrect serializer after UDAF completion Key: SPARK-21277 URL: https://issues.apache.org/jira/browse/SPARK-21277 Project: Spark

[jira] [Commented] (SPARK-21277) Spark is invoking an incorrect serializer after UDAF completion

[ https://issues.apache.org/jira/browse/SPARK-21277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075818#comment-16075818 ] Erik Erlandson commented on SPARK-21277: It would be ideal to document the requirement that all

[jira] [Created] (SPARK-21026) Document jenkins plug-ins assumed by the spark documentation build

Erik Erlandson created SPARK-21026: -- Summary: Document jenkins plug-ins assumed by the spark documentation build Key: SPARK-21026 URL: https://issues.apache.org/jira/browse/SPARK-21026 Project:

[jira] [Commented] (SPARK-10915) Add support for UDAFs in Python

[ https://issues.apache.org/jira/browse/SPARK-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065787#comment-16065787 ] Erik Erlandson commented on SPARK-10915: This would be great for exposing {{TDigest}} aggregation

[jira] [Commented] (SPARK-22647) Docker files for image creation

[ https://issues.apache.org/jira/browse/SPARK-22647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289988#comment-16289988 ] Erik Erlandson commented on SPARK-22647: I'd like to propose migrating our images onto centos,

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477812#comment-16477812 ] Erik Erlandson commented on SPARK-24248: Is the design above using re-sync as the fallback for

[jira] [Commented] (SPARK-24534) Add a way to bypass entrypoint.sh script if no spark cmd is passed

[ https://issues.apache.org/jira/browse/SPARK-24534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511804#comment-16511804 ] Erik Erlandson commented on SPARK-24534: I think this has potential use for customization beyond

[jira] [Resolved] (SPARK-24534) Add a way to bypass entrypoint.sh script if no spark cmd is passed

[ https://issues.apache.org/jira/browse/SPARK-24534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson resolved SPARK-24534. Resolution: Fixed Fix Version/s: 2.4.0 > Add a way to bypass entrypoint.sh script

[jira] [Commented] (SPARK-24091) Internally used ConfigMap prevents use of user-specified ConfigMaps carrying Spark configs files

[ https://issues.apache.org/jira/browse/SPARK-24091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495619#comment-16495619 ] Erik Erlandson commented on SPARK-24091: If we support user-supplied yaml, that may become a

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495615#comment-16495615 ] Erik Erlandson commented on SPARK-24434: Is the template-based solution being explicitly favored

[jira] [Created] (SPARK-24435) Support user-supplied YAML that can be merged with k8s pod descriptions

Erik Erlandson created SPARK-24435: -- Summary: Support user-supplied YAML that can be merged with k8s pod descriptions Key: SPARK-24435 URL: https://issues.apache.org/jira/browse/SPARK-24435 Project:

[jira] [Resolved] (SPARK-24435) Support user-supplied YAML that can be merged with k8s pod descriptions

[ https://issues.apache.org/jira/browse/SPARK-24435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson resolved SPARK-24435. Resolution: Duplicate > Support user-supplied YAML that can be merged with k8s pod

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495642#comment-16495642 ] Erik Erlandson commented on SPARK-24434: [~skonto] given the number of ideas that have gotten

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496991#comment-16496991 ] Erik Erlandson commented on SPARK-24434: [~foxish] is there a technical (or ux) argument for

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498337#comment-16498337 ] Erik Erlandson commented on SPARK-24434: My current take on UX around this feature is that

[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498337#comment-16498337 ] Erik Erlandson edited comment on SPARK-24434 at 6/1/18 5:53 PM: My

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459954#comment-16459954 ] Erik Erlandson commented on SPARK-24135: I think it makes sense to detect these failure states. 

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461154#comment-16461154 ] Erik Erlandson commented on SPARK-24135: IIRC the dynamic allocation heuristic was to avoid

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461324#comment-16461324 ] Erik Erlandson commented on SPARK-24135: > In the case of the executor failing to start at all,

[jira] [Commented] (SPARK-23137) spark.kubernetes.executor.podNamePrefix is ignored

[ https://issues.apache.org/jira/browse/SPARK-23137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329721#comment-16329721 ] Erik Erlandson commented on SPARK-23137: +1, a more general "app prefix" seems more useful >

[jira] [Commented] (SPARK-23324) Announce new Kubernetes back-end for 2.3 release notes

[ https://issues.apache.org/jira/browse/SPARK-23324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351020#comment-16351020 ] Erik Erlandson commented on SPARK-23324: cc [~sameer], [~foxish] > Announce new Kubernetes

[jira] [Created] (SPARK-23324) Announce new Kubernetes back-end for 2.3 release notes

Erik Erlandson created SPARK-23324: -- Summary: Announce new Kubernetes back-end for 2.3 release notes Key: SPARK-23324 URL: https://issues.apache.org/jira/browse/SPARK-23324 Project: Spark

[jira] [Commented] (SPARK-24817) Implement BarrierTaskContext.barrier()

[ https://issues.apache.org/jira/browse/SPARK-24817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567301#comment-16567301 ] Erik Erlandson commented on SPARK-24817: Thanks [~jiangxb] - I'd expect that design to work

[jira] [Commented] (SPARK-24817) Implement BarrierTaskContext.barrier()

[ https://issues.apache.org/jira/browse/SPARK-24817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567358#comment-16567358 ] Erik Erlandson commented on SPARK-24817: I have been looking at the use cases for barrier-mode

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564558#comment-16564558 ] Erik Erlandson commented on SPARK-24615: Am I understanding correctly that this can't assign

[jira] [Commented] (SPARK-24580) List scenarios to be handled by barrier execution mode properly

[ https://issues.apache.org/jira/browse/SPARK-24580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566154#comment-16566154 ] Erik Erlandson commented on SPARK-24580: This is blocking SPARK-24582 which is marked as

[jira] [Commented] (SPARK-24817) Implement BarrierTaskContext.barrier()

[ https://issues.apache.org/jira/browse/SPARK-24817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566159#comment-16566159 ] Erik Erlandson commented on SPARK-24817: I'm curious about what the {{barrier}} invocations

[jira] [Created] (SPARK-25128) multiple simultaneous job submissions against k8s backend cause driver pods to hang

Erik Erlandson created SPARK-25128: -- Summary: multiple simultaneous job submissions against k8s backend cause driver pods to hang Key: SPARK-25128 URL: https://issues.apache.org/jira/browse/SPARK-25128

[jira] [Commented] (SPARK-25128) multiple simultaneous job submissions against k8s backend cause driver pods to hang

[ https://issues.apache.org/jira/browse/SPARK-25128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581537#comment-16581537 ] Erik Erlandson commented on SPARK-25128: [~mcheah], [~liyinan926], wdyt? > multiple

[jira] [Commented] (SPARK-21097) Dynamic allocation will preserve cached data

[ https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594271#comment-16594271 ] Erik Erlandson commented on SPARK-21097: I'm wondering if this is going to be subsumed by the

[jira] [Commented] (SPARK-7768) Make user-defined type (UDT) API public

[ https://issues.apache.org/jira/browse/SPARK-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588934#comment-16588934 ] Erik Erlandson commented on SPARK-7768: --- We use `UserDefinedType`, for example here:

[jira] [Resolved] (SPARK-25275) require memberhip in wheel to run 'su' (in dockerfiles)

[ https://issues.apache.org/jira/browse/SPARK-25275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson resolved SPARK-25275. Resolution: Fixed Fix Version/s: 2.4.0 > require memberhip in wheel to run 'su'

[jira] [Resolved] (SPARK-25287) Check for JIRA_USERNAME and JIRA_PASSWORD up front in merge_spark_pr.py

[ https://issues.apache.org/jira/browse/SPARK-25287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson resolved SPARK-25287. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22294

[jira] [Assigned] (SPARK-25287) Check for JIRA_USERNAME and JIRA_PASSWORD up front in merge_spark_pr.py

[ https://issues.apache.org/jira/browse/SPARK-25287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson reassigned SPARK-25287: -- Assignee: Erik Erlandson > Check for JIRA_USERNAME and JIRA_PASSWORD up front in

[jira] [Commented] (SPARK-25275) require memberhip in wheel to run 'su' (in dockerfiles)

[ https://issues.apache.org/jira/browse/SPARK-25275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597935#comment-16597935 ] Erik Erlandson commented on SPARK-25275: {{merge_spark_pr.py}} failed to close this, closing

[jira] [Created] (SPARK-25287) Check for JIRA_USERNAME and JIRA_PASSWORD up front in merge_spark_pr.py

Erik Erlandson created SPARK-25287: -- Summary: Check for JIRA_USERNAME and JIRA_PASSWORD up front in merge_spark_pr.py Key: SPARK-25287 URL: https://issues.apache.org/jira/browse/SPARK-25287 Project:

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599283#comment-16599283 ] Erik Erlandson commented on SPARK-24434: Stavros, yes, I knew you were working on it, and also

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599299#comment-16599299 ] Erik Erlandson commented on SPARK-24434: To amplify a little from my points above: I co-chair a

[jira] [Updated] (SPARK-25128) multiple simultaneous job submissions against k8s backend cause driver pods to hang

[ https://issues.apache.org/jira/browse/SPARK-25128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-25128: --- Target Version/s: 3.0.0 (was: 2.4.0, 2.3.3) Priority: Minor (was: Major) >

[jira] [Commented] (SPARK-25128) multiple simultaneous job submissions against k8s backend cause driver pods to hang

[ https://issues.apache.org/jira/browse/SPARK-25128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605916#comment-16605916 ] Erik Erlandson commented on SPARK-25128: Retargeting to next release sounds good. There has been

[jira] [Created] (SPARK-25275) require memberhip in wheel to run 'su' (in dockerfiles)

Erik Erlandson created SPARK-25275: -- Summary: require memberhip in wheel to run 'su' (in dockerfiles) Key: SPARK-25275 URL: https://issues.apache.org/jira/browse/SPARK-25275 Project: Spark

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542102#comment-16542102 ] Erik Erlandson commented on SPARK-24793: Also a good point that {{--kill}} and {{--status}} are

[jira] [Comment Edited] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542102#comment-16542102 ] Erik Erlandson edited comment on SPARK-24793 at 7/12/18 7:01 PM: - Also a

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541923#comment-16541923 ] Erik Erlandson commented on SPARK-24793: I am concerned that this is outside the scope of

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541930#comment-16541930 ] Erik Erlandson commented on SPARK-24793: Another possible angle (not mutually exclusive with

[jira] [Commented] (SPARK-23680) entrypoint.sh does not accept arbitrary UIDs, returning as an error

[ https://issues.apache.org/jira/browse/SPARK-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398701#comment-16398701 ] Erik Erlandson commented on SPARK-23680: [~rmartine] thanks for catching this! It will impact

[jira] [Updated] (SPARK-23680) entrypoint.sh does not accept arbitrary UIDs, returning as an error

[ https://issues.apache.org/jira/browse/SPARK-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-23680: --- Flags: Important Labels: easyfix (was: ) > entrypoint.sh does not accept arbitrary

[jira] [Updated] (SPARK-23891) Debian based Dockerfile

[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-23891: --- Priority: Minor (was: Major) Issue Type: New Feature (was: Bug) > Debian based

[jira] [Commented] (SPARK-23891) Debian based Dockerfile

[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436185#comment-16436185 ] Erik Erlandson commented on SPARK-23891: The question of what OS base to use for "canonical"

[jira] [Commented] (SPARK-23891) Debian based Dockerfile

[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436196#comment-16436196 ] Erik Erlandson commented on SPARK-23891: I do think that these reports are very useful for

[jira] [Commented] (SPARK-23891) Debian based Dockerfile

[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438410#comment-16438410 ] Erik Erlandson commented on SPARK-23891: [~SercanKaraoglu] thanks for the information! You are

[jira] [Resolved] (SPARK-23680) entrypoint.sh does not accept arbitrary UIDs, returning as an error

[ https://issues.apache.org/jira/browse/SPARK-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson resolved SPARK-23680. Resolution: Fixed Target Version/s: 2.3.1, 2.4.0 merged to master >

[jira] [Commented] (SPARK-23680) entrypoint.sh does not accept arbitrary UIDs, returning as an error

[ https://issues.apache.org/jira/browse/SPARK-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402266#comment-16402266 ] Erik Erlandson commented on SPARK-23680: commit workflow indicates to set the Assignee, however I

[jira] [Updated] (SPARK-25782) Add PCA Aggregator to support grouping

[ https://issues.apache.org/jira/browse/SPARK-25782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-25782: --- Target Version/s: 3.0.0 Component/s: ML Issue Type: New Feature (was:

  1   2   >