[jira] [Assigned] (LIVY-539) Incorrect default value for livy.rsc.launcher.port.range in livy-client.conf

[ https://issues.apache.org/jira/browse/LIVY-539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-539: --- Assignee: Adrian Petrescu > Incorrect default value for livy.rsc.launcher.port.range in l

[jira] [Resolved] (LIVY-539) Incorrect default value for livy.rsc.launcher.port.range in livy-client.conf

[ https://issues.apache.org/jira/browse/LIVY-539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-539. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 132 [https

[jira] [Assigned] (SPARK-26119) Task metrics summary in the stage page should contain only successful tasks metrics

[ https://issues.apache.org/jira/browse/SPARK-26119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26119: -- Assignee: shahid > Task metrics summary in the stage page should contain o

[jira] [Resolved] (SPARK-26119) Task metrics summary in the stage page should contain only successful tasks metrics

[ https://issues.apache.org/jira/browse/SPARK-26119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26119. Resolution: Fixed Fix Version/s: 2.4.1 3.0.0 Issue resolved

[jira] [Assigned] (SPARK-26083) Pyspark command is not working properly with default Docker Image build

[ https://issues.apache.org/jira/browse/SPARK-26083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26083: -- Assignee: Qi Shao > Pyspark command is not working properly with default Doc

[jira] [Resolved] (SPARK-26083) Pyspark command is not working properly with default Docker Image build

[ https://issues.apache.org/jira/browse/SPARK-26083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26083. Resolution: Fixed Fix Version/s: (was: 2.4.1) 3.0.0 Issue

[jira] [Resolved] (LIVY-533) Spark jobs submitted via programmatic API cannot always be canceled

[ https://issues.apache.org/jira/browse/LIVY-533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-533. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 128 [https

[jira] [Updated] (SPARK-26219) Executor summary is not getting updated for failure jobs in history server UI

[ https://issues.apache.org/jira/browse/SPARK-26219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26219: --- Fix Version/s: 2.4.1 > Executor summary is not getting updated for failure jobs in hist

[jira] [Updated] (SPARK-26256) Add proper labels when deleting pods

[ https://issues.apache.org/jira/browse/SPARK-26256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26256: --- Fix Version/s: 2.4.1 > Add proper labels when deleting p

[jira] [Assigned] (SPARK-26256) Add proper labels when deleting pods

[ https://issues.apache.org/jira/browse/SPARK-26256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26256: -- Assignee: Stavros Kontopoulos > Add proper labels when deleting p

[jira] [Resolved] (SPARK-26256) Add proper labels when deleting pods

[ https://issues.apache.org/jira/browse/SPARK-26256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26256. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23209 [https

[jira] [Resolved] (LIVY-502) Cleanup Hive dependencies

[ https://issues.apache.org/jira/browse/LIVY-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-502. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 117 [https

[jira] [Assigned] (LIVY-502) Cleanup Hive dependencies

[ https://issues.apache.org/jira/browse/LIVY-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-502: --- Assignee: Marco Gaido > Cleanup Hive dependenc

[jira] [Resolved] (SPARK-26219) Executor summary is not getting updated for failure jobs in history server UI

[ https://issues.apache.org/jira/browse/SPARK-26219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26219. Resolution: Fixed Assignee: shahid Fix Version/s: 3.0.0 > Execu

[jira] [Commented] (SPARK-23078) Allow Submitting Spark Thrift Server in Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-23078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705282#comment-16705282 ] Marcelo Vanzin commented on SPARK-23078: Is this actually needed now that k8s supports client

[jira] [Commented] (SPARK-26239) Add configurable auth secret source in k8s backend

[ https://issues.apache.org/jira/browse/SPARK-26239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705277#comment-16705277 ] Marcelo Vanzin commented on SPARK-26239: That can work but it doesn't address the 3rd bullet

[jira] [Created] (SPARK-26239) Add configurable auth secret source in k8s backend

Marcelo Vanzin created SPARK-26239: -- Summary: Add configurable auth secret source in k8s backend Key: SPARK-26239 URL: https://issues.apache.org/jira/browse/SPARK-26239 Project: Spark Issue

[jira] [Commented] (SPARK-26043) Make SparkHadoopUtil private to Spark

[ https://issues.apache.org/jira/browse/SPARK-26043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705045#comment-16705045 ] Marcelo Vanzin commented on SPARK-26043: It would be better if you can show what you mean

[jira] [Resolved] (SPARK-25501) Kafka delegation token support

[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25501. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22598 [https

[jira] [Assigned] (SPARK-25501) Kafka delegation token support

[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25501: -- Assignee: Gabor Somogyi > Kafka delegation token supp

[jira] [Assigned] (LIVY-503) Move RPC classes used in thrifserver in a separate module

[ https://issues.apache.org/jira/browse/LIVY-503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-503: --- Assignee: Marcelo Vanzin > Move RPC classes used in thrifserver in a separate mod

[jira] [Resolved] (LIVY-503) Move RPC classes used in thrifserver in a separate module

[ https://issues.apache.org/jira/browse/LIVY-503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-503. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 118 [https

[jira] [Assigned] (SPARK-26015) Include a USER directive in project provided Spark Dockerfiles

[ https://issues.apache.org/jira/browse/SPARK-26015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26015: -- Assignee: Rob Vesse > Include a USER directive in project provided Spark Dockerfi

[jira] [Resolved] (SPARK-26015) Include a USER directive in project provided Spark Dockerfiles

[ https://issues.apache.org/jira/browse/SPARK-26015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26015. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23017 [https

[jira] [Assigned] (SPARK-26184) Last updated time is not getting updated in the History Server UI

[ https://issues.apache.org/jira/browse/SPARK-26184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26184: -- Assignee: shahid > Last updated time is not getting updated in the History Server

[jira] [Resolved] (SPARK-26184) Last updated time is not getting updated in the History Server UI

[ https://issues.apache.org/jira/browse/SPARK-26184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26184. Resolution: Fixed Fix Version/s: 2.4.1 3.0.0 Issue resolved

[jira] [Resolved] (SPARK-26186) In progress applications with last updated time is lesser than the cleaning interval are getting removed during cleaning logs

[ https://issues.apache.org/jira/browse/SPARK-26186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26186. Resolution: Fixed Fix Version/s: 2.4.1 3.0.0 Issue resolved

[jira] [Assigned] (SPARK-26186) In progress applications with last updated time is lesser than the cleaning interval are getting removed during cleaning logs

[ https://issues.apache.org/jira/browse/SPARK-26186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26186: -- Assignee: shahid > In progress applications with last updated time is lesser t

[jira] [Resolved] (LIVY-537) Add numExecutor configuration when creating session in thriftserver

[ https://issues.apache.org/jira/browse/LIVY-537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-537. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 130 [https

[jira] [Created] (SPARK-26194) Support automatic spark.authenticate secret in Kubernetes backend

Marcelo Vanzin created SPARK-26194: -- Summary: Support automatic spark.authenticate secret in Kubernetes backend Key: SPARK-26194 URL: https://issues.apache.org/jira/browse/SPARK-26194 Project: Spark

[jira] [Resolved] (SPARK-24219) Improve the docker build script to avoid copying everything in example

[ https://issues.apache.org/jira/browse/SPARK-24219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24219. Resolution: Duplicate I fixed this as part of SPARK-26025. > Improve the docker bu

[jira] [Resolved] (SPARK-24383) spark on k8s: "driver-svc" are not getting deleted

[ https://issues.apache.org/jira/browse/SPARK-24383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24383. Resolution: Not A Problem This has been working reliably for me. If your k8s server

[jira] [Resolved] (SPARK-24577) Spark submit fails with documentation example spark-pi

[ https://issues.apache.org/jira/browse/SPARK-24577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24577. Resolution: Duplicate > Spark submit fails with documentation example spark

[jira] [Resolved] (SPARK-24600) Improve support for building different types of images in dockerfile

[ https://issues.apache.org/jira/browse/SPARK-24600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24600. Resolution: Duplicate > Improve support for building different types of ima

[jira] [Resolved] (SPARK-26096) k8s integration tests should run R tests

[ https://issues.apache.org/jira/browse/SPARK-26096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26096. Resolution: Duplicate > k8s integration tests should run R te

[jira] [Commented] (SPARK-26125) Delegation Token seems not appropriately stored on secrets of Kubernetes/Kerberized HDFS

[ https://issues.apache.org/jira/browse/SPARK-26125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701200#comment-16701200 ] Marcelo Vanzin commented on SPARK-26125: Pretty sure this works with my patch for SPARK-25815

[jira] [Resolved] (SPARK-25744) Allow kubernetes integration tests to be run against a real cluster.

[ https://issues.apache.org/jira/browse/SPARK-25744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25744. Resolution: Duplicate > Allow kubernetes integration tests to be run against a r

[jira] [Resolved] (SPARK-26064) Unable to fetch jar from remote repo while running spark-submit on kubernetes

[ https://issues.apache.org/jira/browse/SPARK-26064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26064. Resolution: Invalid I'm closing this for the time being. If you have a question please

[jira] [Resolved] (SPARK-26190) SparkLauncher: Allow users to set their own submitter script instead of hardcoded spark-submit

[ https://issues.apache.org/jira/browse/SPARK-26190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26190. Resolution: Won't Fix I'm closing this for now until I see a better use case. Seems

[jira] [Commented] (SPARK-26190) SparkLauncher: Allow users to set their own submitter script instead of hardcoded spark-submit

[ https://issues.apache.org/jira/browse/SPARK-26190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701140#comment-16701140 ] Marcelo Vanzin commented on SPARK-26190: bq. Can you give me one good reason that why you must

[jira] [Created] (LIVY-538) Add integration tests for the thrift server

Marcelo Vanzin created LIVY-538: --- Summary: Add integration tests for the thrift server Key: LIVY-538 URL: https://issues.apache.org/jira/browse/LIVY-538 Project: Livy Issue Type: Sub-task

[jira] [Commented] (SPARK-26190) SparkLauncher: Allow users to set their own submitter script instead of hardcoded spark-submit

[ https://issues.apache.org/jira/browse/SPARK-26190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700937#comment-16700937 ] Marcelo Vanzin commented on SPARK-26190: If you need to run things before spark-submit runs

[jira] [Commented] (SPARK-26190) SparkLauncher: Allow users to set their own submitter script instead of hardcoded spark-submit

[ https://issues.apache.org/jira/browse/SPARK-26190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700896#comment-16700896 ] Marcelo Vanzin commented on SPARK-26190: Just run your script without SparkLauncher? e.g. using

[jira] [Commented] (SPARK-26190) SparkLauncher: Allow users to set their own submitter script instead of hardcoded spark-submit

[ https://issues.apache.org/jira/browse/SPARK-26190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700890#comment-16700890 ] Marcelo Vanzin commented on SPARK-26190: Based solely on what you wrote here, I'm leaning

[jira] [Assigned] (SPARK-26025) Docker image build on dev builds is slow

[ https://issues.apache.org/jira/browse/SPARK-26025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26025: -- Assignee: Marcelo Vanzin > Docker image build on dev builds is s

[jira] [Resolved] (SPARK-26025) Docker image build on dev builds is slow

[ https://issues.apache.org/jira/browse/SPARK-26025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26025. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23019 [https

Re: [crypto] Updated to Support OpenSSL 1.1.0i

Hi Alex, There have been other attempts at this (e.g. https://github.com/apache/commons-crypto/pull/90), and my main concern with those is that they require different builds for OpenSSL 1.0 and 1.1. It seems your approach falls in that bucket, at least without additional work. The problem with

[jira] [Resolved] (SPARK-25451) Stages page doesn't show the right number of the total tasks

[ https://issues.apache.org/jira/browse/SPARK-25451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25451. Resolution: Fixed Assignee: shahid Fix Version/s: 3.0.0

[jira] [Resolved] (SPARK-26100) [History server ]Jobs table and Aggregate metrics table are showing lesser number of tasks

[ https://issues.apache.org/jira/browse/SPARK-26100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26100. Resolution: Fixed Assignee: shahid Fix Version/s: 3.0.0

[jira] [Resolved] (SPARK-25960) Support subpath mounting with Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-25960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25960. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23026 [https

[jira] [Assigned] (SPARK-25960) Support subpath mounting with Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-25960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25960: -- Assignee: Nihar Sheth > Support subpath mounting with Kuberne

[jira] [Resolved] (LIVY-535) Config option “livy.server.session.max-creation” doesn't work properly

[ https://issues.apache.org/jira/browse/LIVY-535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-535. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 129 [https

[jira] [Commented] (SPARK-26103) OutOfMemory error with large query plans

[ https://issues.apache.org/jira/browse/SPARK-26103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699239#comment-16699239 ] Marcelo Vanzin commented on SPARK-26103: The proposed patch would probably also fix SPARK-25380

[jira] [Resolved] (SPARK-26095) make-distribution.sh is hanging in jenkins

[ https://issues.apache.org/jira/browse/SPARK-26095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26095. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23061 [https

[jira] [Assigned] (SPARK-26095) make-distribution.sh is hanging in jenkins

[ https://issues.apache.org/jira/browse/SPARK-26095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26095: -- Assignee: Marcelo Vanzin > make-distribution.sh is hanging in jenk

[jira] [Created] (SPARK-26096) k8s integration tests should run R tests

Marcelo Vanzin created SPARK-26096: -- Summary: k8s integration tests should run R tests Key: SPARK-26096 URL: https://issues.apache.org/jira/browse/SPARK-26096 Project: Spark Issue Type

Re: Make Scala 2.12 as default Scala version in Spark 3.0

Now that the switch to 2.12 by default has been made, it might be good to have a serious discussion about dropping 2.11 altogether. Many of the main arguments have already been talked about. But I don't remember anyone mentioning how easy it would be to break the 2.11 build now. For example, the

[jira] [Created] (SPARK-26095) make-distribution.sh is hanging in jenkins

Marcelo Vanzin created SPARK-26095: -- Summary: make-distribution.sh is hanging in jenkins Key: SPARK-26095 URL: https://issues.apache.org/jira/browse/SPARK-26095 Project: Spark Issue Type

[jira] [Created] (SPARK-26079) Flaky test: StreamingQueryListenersConfSuite

Marcelo Vanzin created SPARK-26079: -- Summary: Flaky test: StreamingQueryListenersConfSuite Key: SPARK-26079 URL: https://issues.apache.org/jira/browse/SPARK-26079 Project: Spark Issue Type

[jira] [Comment Edited] (SPARK-25957) Skip building spark-r docker image if spark distribution does not have R support

[ https://issues.apache.org/jira/browse/SPARK-25957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687306#comment-16687306 ] Marcelo Vanzin edited comment on SPARK-25957 at 11/15/18 12:16 AM: --- I

[jira] [Commented] (SPARK-25957) Skip building spark-r docker image if spark distribution does not have R support

[ https://issues.apache.org/jira/browse/SPARK-25957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687306#comment-16687306 ] Marcelo Vanzin commented on SPARK-25957: I prefer to keep the current behavior and add options

[jira] [Assigned] (SPARK-25778) WriteAheadLogBackedBlockRDD in YARN Cluster Mode Fails due lack of access to tmpDir from $PWD to HDFS

[ https://issues.apache.org/jira/browse/SPARK-25778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25778: -- Assignee: Greg Senia > WriteAheadLogBackedBlockRDD in YARN Cluster Mode Fails

[jira] [Resolved] (SPARK-25778) WriteAheadLogBackedBlockRDD in YARN Cluster Mode Fails due lack of access to tmpDir from $PWD to HDFS

[ https://issues.apache.org/jira/browse/SPARK-25778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25778. Resolution: Fixed Fix Version/s: 2.4.1 3.0.0 Issue resolved

[jira] [Resolved] (SPARK-23067) Allow for easier debugging of the docker container

[ https://issues.apache.org/jira/browse/SPARK-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23067. Resolution: Duplicate This was added in SPARK-24534. Yay for searching before filing

[jira] [Resolved] (SPARK-25118) Need a solution to persist Spark application console outputs when running in shell/yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-25118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25118. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22504 [https

[jira] [Assigned] (SPARK-25118) Need a solution to persist Spark application console outputs when running in shell/yarn client mode

[ https://issues.apache.org/jira/browse/SPARK-25118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25118: -- Assignee: Ankur Gupta > Need a solution to persist Spark application console outp

Re: which classes/methods are considered as private in Spark?

On Tue, Nov 13, 2018 at 6:26 PM Wenchen Fan wrote: > Recently I updated the MiMa exclusion rules, and found MiMa tracks some > private classes/methods unexpectedly. Could you clarify what you mean here? Mima has some known limitations such as not handling "private[blah]" very well (because that

[jira] [Updated] (SPARK-26048) Flume connector for Spark 2.4 does not exist in Maven repository

[ https://issues.apache.org/jira/browse/SPARK-26048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26048: --- Target Version/s: 2.4.1 Priority: Blocker (was: Major) > Flume connec

[jira] [Commented] (SPARK-25598) Remove flume connector in Spark 3

[ https://issues.apache.org/jira/browse/SPARK-25598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685841#comment-16685841 ] Marcelo Vanzin commented on SPARK-25598: Hmm, seems that we missed that one. Can you file

[jira] [Created] (SPARK-26043) Make SparkHadoopUtil private to Spark

Marcelo Vanzin created SPARK-26043: -- Summary: Make SparkHadoopUtil private to Spark Key: SPARK-26043 URL: https://issues.apache.org/jira/browse/SPARK-26043 Project: Spark Issue Type

[jira] [Commented] (SPARK-25924) k8s integration tests on jenkins do not save log output

[ https://issues.apache.org/jira/browse/SPARK-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685562#comment-16685562 ] Marcelo Vanzin commented on SPARK-25924: Hey [~shaneknapp], you maked this as fixed but I still

[jira] [Created] (SPARK-26025) Docker image build on dev builds is slow

Marcelo Vanzin created SPARK-26025: -- Summary: Docker image build on dev builds is slow Key: SPARK-26025 URL: https://issues.apache.org/jira/browse/SPARK-26025 Project: Spark Issue Type

[jira] [Assigned] (SPARK-25897) Cannot run k8s integration tests in sbt

[ https://issues.apache.org/jira/browse/SPARK-25897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25897: -- Assignee: Marcelo Vanzin > Cannot run k8s integration tests in

Re: [ANNOUNCE] Announcing Apache Spark 2.4.0

+user@ >> -- Forwarded message - >> From: Wenchen Fan >> Date: Thu, Nov 8, 2018 at 10:55 PM >> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >> To: Spark dev list >> >> >> Hi all, >> >> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release adds >> Barrier

Re: [ANNOUNCE] Announcing Apache Spark 2.4.0

+user@ >> -- Forwarded message - >> From: Wenchen Fan >> Date: Thu, Nov 8, 2018 at 10:55 PM >> Subject: [ANNOUNCE] Announcing Apache Spark 2.4.0 >> To: Spark dev list >> >> >> Hi all, >> >> Apache Spark 2.4.0 is the fifth release in the 2.x line. This release adds >> Barrier

[jira] [Resolved] (SPARK-25897) Cannot run k8s integration tests in sbt

[ https://issues.apache.org/jira/browse/SPARK-25897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25897. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22909 [https

[jira] [Assigned] (LIVY-518) Add support for Spark 2.4

[ https://issues.apache.org/jira/browse/LIVY-518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-518: --- Assignee: Hyukjin Kwon > Add support for Spark

Re: Test and support only LTS JDK release?

https://www.oracle.com/technetwork/java/javase/eol-135779.html On Tue, Nov 6, 2018 at 2:56 PM Felix Cheung wrote: > > Is there a list of LTS release that I can reference? > > > > From: Ryan Blue > Sent: Tuesday, November 6, 2018 1:28 PM > To: sn...@snazy.de > Cc:

[jira] [Updated] (SPARK-25222) Spark on Kubernetes Pod Watcher dumps raw container status

[ https://issues.apache.org/jira/browse/SPARK-25222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-25222: --- Fix Version/s: 3.0.0 > Spark on Kubernetes Pod Watcher dumps raw container sta

[jira] [Assigned] (SPARK-25222) Spark on Kubernetes Pod Watcher dumps raw container status

[ https://issues.apache.org/jira/browse/SPARK-25222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25222: -- Assignee: Rob Vesse > Spark on Kubernetes Pod Watcher dumps raw container sta

Re: Test and support only LTS JDK release?

+1, that's always been my view. Although, to be fair, and as Sean mentioned, the jump from jdk8 is probably the harder part. After that it's less likely (hopefully?) that we'll run into issues in non-LTS releases. And even if we don't officially support them, trying to keep up with breaking

[jira] [Resolved] (SPARK-25871) Streaming WAL should not use hdfs erasure coding, regardless of FS defaults

[ https://issues.apache.org/jira/browse/SPARK-25871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25871. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22882 [https

[jira] [Assigned] (SPARK-25871) Streaming WAL should not use hdfs erasure coding, regardless of FS defaults

[ https://issues.apache.org/jira/browse/SPARK-25871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25871: -- Assignee: Imran Rashid > Streaming WAL should not use hdfs erasure cod

[jira] [Assigned] (LIVY-525) Response Content-Type is application/json, but a simple string gets returned

[ https://issues.apache.org/jira/browse/LIVY-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned LIVY-525: --- Assignee: Ingo Schuster > Response Content-Type is application/json, but a simple string g

[jira] [Commented] (SPARK-25590) kubernetes-model-2.0.0.jar masks default Spark logging config

[ https://issues.apache.org/jira/browse/SPARK-25590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673689#comment-16673689 ] Marcelo Vanzin commented on SPARK-25590: {noformat} $ jar tf assembly/target/scala-2.11/jars

[jira] [Assigned] (SPARK-25827) Replicating a block > 2gb with encryption fails

[ https://issues.apache.org/jira/browse/SPARK-25827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25827: -- Assignee: Imran Rashid > Replicating a block > 2gb with encryption

[jira] [Resolved] (SPARK-25827) Replicating a block > 2gb with encryption fails

[ https://issues.apache.org/jira/browse/SPARK-25827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25827. Resolution: Fixed Fix Version/s: 2.4.1 3.0.0 Issue resolved

[jira] [Commented] (SPARK-25924) k8s integration tests on jenkins do not save log output

[ https://issues.apache.org/jira/browse/SPARK-25924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673431#comment-16673431 ] Marcelo Vanzin commented on SPARK-25924: [~shaneknapp], it seems that the ITs do generate

[jira] [Created] (SPARK-25924) k8s integration tests on jenkins do not save log output

Marcelo Vanzin created SPARK-25924: -- Summary: k8s integration tests on jenkins do not save log output Key: SPARK-25924 URL: https://issues.apache.org/jira/browse/SPARK-25924 Project: Spark

[jira] [Created] (SPARK-25920) Avoid custom processing of CLI options for cluster submission

Marcelo Vanzin created SPARK-25920: -- Summary: Avoid custom processing of CLI options for cluster submission Key: SPARK-25920 URL: https://issues.apache.org/jira/browse/SPARK-25920 Project: Spark

[jira] [Resolved] (SPARK-25912) Starting history server throwing exception, when enable the local store directory

[ https://issues.apache.org/jira/browse/SPARK-25912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25912. Resolution: Not A Problem > Starting history server throwing exception, when ena

[jira] [Commented] (SPARK-25912) Starting history server throwing exception, when enable the local store directory

[ https://issues.apache.org/jira/browse/SPARK-25912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672043#comment-16672043 ] Marcelo Vanzin commented on SPARK-25912: No. As I said, that is how directories work. If you

[jira] [Commented] (SPARK-25912) Starting history server throwing exception, when enable the local store directory

[ https://issues.apache.org/jira/browse/SPARK-25912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672006#comment-16672006 ] Marcelo Vanzin commented on SPARK-25912: That's how directories work. Still don't see an issue

[jira] [Commented] (SPARK-25912) Starting history server throwing exception, when enable the local store directory

[ https://issues.apache.org/jira/browse/SPARK-25912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671966#comment-16671966 ] Marcelo Vanzin commented on SPARK-25912: Sorry, what's the bug here? You created an unwritable

[jira] [Assigned] (SPARK-25837) Web UI does not respect spark.ui.retainedJobs in some instances

[ https://issues.apache.org/jira/browse/SPARK-25837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25837: -- Assignee: Patrick Brown (was: Apache Spark) > Web UI does not resp

[jira] [Resolved] (SPARK-25837) Web UI does not respect spark.ui.retainedJobs in some instances

[ https://issues.apache.org/jira/browse/SPARK-25837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25837. Resolution: Fixed Fix Version/s: 2.4.1 3.0.0

[jira] [Commented] (SPARK-20608) Standby namenodes should be allowed to included in yarn.spark.access.namenodes to support HDFS HA

[ https://issues.apache.org/jira/browse/SPARK-20608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671828#comment-16671828 ] Marcelo Vanzin commented on SPARK-20608: That error doesn't provide a lot of information. Maybe

[jira] [Created] (SPARK-25903) Flaky test: BarrierTaskContextSuite.throw exception on barrier() call timeout

Marcelo Vanzin created SPARK-25903: -- Summary: Flaky test: BarrierTaskContextSuite.throw exception on barrier() call timeout Key: SPARK-25903 URL: https://issues.apache.org/jira/browse/SPARK-25903

Re: [VOTE] SPARK 2.4.0 (RC5)

+1 On Mon, Oct 29, 2018 at 3:22 AM Wenchen Fan wrote: > > Please vote on releasing the following candidate as Apache Spark version > 2.4.0. > > The vote is open until November 1 PST and passes if a majority +1 PMC votes > are cast, with > a minimum of 3 +1 votes. > > [ ] +1 Release this package

[jira] [Resolved] (SPARK-25855) Don't use Erasure Coding for event log files

[ https://issues.apache.org/jira/browse/SPARK-25855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25855. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22881 [https

<    3   4   5   6   7   8   9   10   11   12   >